+ All Categories
Home > Documents > Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key...

Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key...

Date post: 01-Nov-2018
Category:
Upload: vanngoc
View: 238 times
Download: 2 times
Share this document with a friend
43
Cyme Documentation Release 0.0.4 VMWare, Inc. March 20, 2012
Transcript
Page 1: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme DocumentationRelease 004

VMWare Inc

March 20 2012

CONTENTS

i

ii

Cyme Documentation Release 004

Contents

CONTENTS 1

Cyme Documentation Release 004

2 CONTENTS

CHAPTER

ONE

INTRODUCTION

bull Synopsisbull Requirementsbull Getting startedbull API Reference

ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale

bull Componentsndash Programsndash Models

App Broker Instance Queue

bull Supervisorbull Controllerbull HTTP

11 Synopsis

Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service

12 Requirements

bull Python 26 or later

bull cl

bull eventlet

3

Cyme Documentation Release 004

bull django

bull django-celery

Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios

13 Getting started

You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch

Start one or more branches

$ cyme-branch 8001 -i branch1 -D cymebranch1

$ cyme-branch 8002 -i branch2 -D cymebranch2

The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory

Create a new application named foo

$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked

name foo broker password guesthostname 127001userid guestport 5672virtual_host

Note that we can edit the broker connection details here by using a POST request

$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo

Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed

Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data

Create a new Celery worker instance

$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked

is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712

4 Chapter 1 Introduction

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 2: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

CONTENTS

i

ii

Cyme Documentation Release 004

Contents

CONTENTS 1

Cyme Documentation Release 004

2 CONTENTS

CHAPTER

ONE

INTRODUCTION

bull Synopsisbull Requirementsbull Getting startedbull API Reference

ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale

bull Componentsndash Programsndash Models

App Broker Instance Queue

bull Supervisorbull Controllerbull HTTP

11 Synopsis

Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service

12 Requirements

bull Python 26 or later

bull cl

bull eventlet

3

Cyme Documentation Release 004

bull django

bull django-celery

Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios

13 Getting started

You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch

Start one or more branches

$ cyme-branch 8001 -i branch1 -D cymebranch1

$ cyme-branch 8002 -i branch2 -D cymebranch2

The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory

Create a new application named foo

$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked

name foo broker password guesthostname 127001userid guestport 5672virtual_host

Note that we can edit the broker connection details here by using a POST request

$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo

Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed

Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data

Create a new Celery worker instance

$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked

is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712

4 Chapter 1 Introduction

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 3: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

ii

Cyme Documentation Release 004

Contents

CONTENTS 1

Cyme Documentation Release 004

2 CONTENTS

CHAPTER

ONE

INTRODUCTION

bull Synopsisbull Requirementsbull Getting startedbull API Reference

ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale

bull Componentsndash Programsndash Models

App Broker Instance Queue

bull Supervisorbull Controllerbull HTTP

11 Synopsis

Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service

12 Requirements

bull Python 26 or later

bull cl

bull eventlet

3

Cyme Documentation Release 004

bull django

bull django-celery

Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios

13 Getting started

You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch

Start one or more branches

$ cyme-branch 8001 -i branch1 -D cymebranch1

$ cyme-branch 8002 -i branch2 -D cymebranch2

The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory

Create a new application named foo

$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked

name foo broker password guesthostname 127001userid guestport 5672virtual_host

Note that we can edit the broker connection details here by using a POST request

$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo

Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed

Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data

Create a new Celery worker instance

$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked

is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712

4 Chapter 1 Introduction

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 4: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

Contents

CONTENTS 1

Cyme Documentation Release 004

2 CONTENTS

CHAPTER

ONE

INTRODUCTION

bull Synopsisbull Requirementsbull Getting startedbull API Reference

ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale

bull Componentsndash Programsndash Models

App Broker Instance Queue

bull Supervisorbull Controllerbull HTTP

11 Synopsis

Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service

12 Requirements

bull Python 26 or later

bull cl

bull eventlet

3

Cyme Documentation Release 004

bull django

bull django-celery

Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios

13 Getting started

You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch

Start one or more branches

$ cyme-branch 8001 -i branch1 -D cymebranch1

$ cyme-branch 8002 -i branch2 -D cymebranch2

The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory

Create a new application named foo

$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked

name foo broker password guesthostname 127001userid guestport 5672virtual_host

Note that we can edit the broker connection details here by using a POST request

$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo

Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed

Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data

Create a new Celery worker instance

$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked

is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712

4 Chapter 1 Introduction

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 5: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

2 CONTENTS

CHAPTER

ONE

INTRODUCTION

bull Synopsisbull Requirementsbull Getting startedbull API Reference

ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale

bull Componentsndash Programsndash Models

App Broker Instance Queue

bull Supervisorbull Controllerbull HTTP

11 Synopsis

Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service

12 Requirements

bull Python 26 or later

bull cl

bull eventlet

3

Cyme Documentation Release 004

bull django

bull django-celery

Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios

13 Getting started

You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch

Start one or more branches

$ cyme-branch 8001 -i branch1 -D cymebranch1

$ cyme-branch 8002 -i branch2 -D cymebranch2

The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory

Create a new application named foo

$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked

name foo broker password guesthostname 127001userid guestport 5672virtual_host

Note that we can edit the broker connection details here by using a POST request

$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo

Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed

Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data

Create a new Celery worker instance

$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked

is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712

4 Chapter 1 Introduction

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 6: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

CHAPTER

ONE

INTRODUCTION

bull Synopsisbull Requirementsbull Getting startedbull API Reference

ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale

bull Componentsndash Programsndash Models

App Broker Instance Queue

bull Supervisorbull Controllerbull HTTP

11 Synopsis

Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service

12 Requirements

bull Python 26 or later

bull cl

bull eventlet

3

Cyme Documentation Release 004

bull django

bull django-celery

Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios

13 Getting started

You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch

Start one or more branches

$ cyme-branch 8001 -i branch1 -D cymebranch1

$ cyme-branch 8002 -i branch2 -D cymebranch2

The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory

Create a new application named foo

$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked

name foo broker password guesthostname 127001userid guestport 5672virtual_host

Note that we can edit the broker connection details here by using a POST request

$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo

Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed

Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data

Create a new Celery worker instance

$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked

is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712

4 Chapter 1 Introduction

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 7: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

bull django

bull django-celery

Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios

13 Getting started

You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch

Start one or more branches

$ cyme-branch 8001 -i branch1 -D cymebranch1

$ cyme-branch 8002 -i branch2 -D cymebranch2

The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory

Create a new application named foo

$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked

name foo broker password guesthostname 127001userid guestport 5672virtual_host

Note that we can edit the broker connection details here by using a POST request

$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo

Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed

Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data

Create a new Celery worker instance

$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked

is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712

4 Chapter 1 Introduction

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 8: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

queues []broker password guest

userid guesthostname 127001virtual_host port 5672

max_concurrency 1min_concurrency 1

Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin

In the logs of the affected branch you should now see something like this

582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712

-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672

celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=

rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo

rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo

rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1

Now that we have created an instance we can list the available instances

$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked

[a35f2518-13bb-4403-bbdf-dd8751077712]

Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP

Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)

13 Getting started 5

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 9: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

Create the queue by performing the following request

$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked

exchange t2routing_key t2options nullname t2exchange_type null

The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches

$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked

[tasks]

Now we can make our worker instance consume from the tasks queue to process tasks sent to it

$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2

HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked

ok ok

In the logs for the branch that this is instance is a member of you should now see

[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks

If the test was successful you can clean up after yourself by

bull Cancelling consuming from the tasks queue

$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks

bull Deleting the tasks queue

$ curl -X DELETE -i httplocalhost8001fooqueues

bull and finally deleting the worker instance

$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712

The worker instance should now be shutdown by the branch supervisor

6 Chapter 1 Introduction

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 10: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

14 API Reference

141 Applications

bull Create new named application

[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str

If hostname is not provided then any other broker parameters will be ignored and the default broker will be used

bull List all available applications

GET httpbranchport

bull Get the configuration for app by name

GET httpbranchportname

142 Instances

bull Create and start an anonymous instance associated with app

[PUT|POST] httpbranchportltappgtinstances

This will return the details of the new id including the instance name (which for anonymous instances is an UUID)

bull Create and start a named instance associated with app

[PUT|POST] httpbranchportltappgtinstancesltnamegt

bull List all available instances associated with an app

GET httpbranchportltappgt

bull Get the details of an instance by name

GET httpbranchportltappgtinstancesltnamegt

bull Delete an instance by name

DELETE httpbranchportltappgtinstancesltnamegt

143 Queues

bull Create a new queue declaration by name

[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict

14 API Reference 7

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 11: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()

bull Get the declaration for a queue by name

GET httpbranchportltappgtqueuesltnamegt

bull Get a list of available queues

GET httpbranchportltappgtqueues

144 Consumers

Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name

bull Tell an instance by name to consume from queue by name

[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

bull Tell an instance by name to stop consuming from queue by name

DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt

145 Queueing Tasks

Queueing an URL will result in one of the worker instances to execute that request as soon as possible

[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data

The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded

When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker

Examples

GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133

POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries

146 Querying Task State

bull To get the current state of a task

GET httpbranchportltappgtqueryltuuidgtstate

bull To get the return value of a task

8 Chapter 1 Introduction

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 12: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

GET httpbranchportltappgtqueryltuuidgtresult

bull To wait for a task to complete and return its result

GET httpbranchportltappgtqueryltuuidgtwait

147 Instance details and statistics

To get configuration details and statistics for a particular instance

GET httpbranchportltappgtinstanceltnamegtstats

148 Autoscale

bull To set the maxmin concurrency settings of an instance

POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int

bull To get the maxmin concurrency settings of an instance

GET httpbranchportltappgtinstanceltnamegtautoscale

15 Components

151 Programs

bull cyme ltcymemanagementcommandscyme

This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments

bull cyme-branch

Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments

152 Models

The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)

App

see cymemodelsApp

Every instance belongs to an application and the application contains the default broker configuration

15 Components 9

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 13: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

Broker

see cymemodelsBroker

The connection parameters for a specific broker (hostname port userid password virtual_host)

Instance

see cymemodelsInstance

This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)

Queue

see cymemodelsQueue

A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()

16 Supervisor

see cymesupervisor

The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync

It is responsible for

bull Stopping removed instances

bull Starting new instances

bull Restarting unresponsivekilled instances

bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies

bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)

17 Controller

see cymecontroller

The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly

10 Chapter 1 Introduction

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 14: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

18 HTTP

The http server currently serves up an admin instance where you can add remove and modify instances

The http server can be disabled using the --without-http option

18 HTTP 11

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 15: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

12 Chapter 1 Introduction

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 16: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

CHAPTER

TWO

CHANGE HISTORY

bull 001

21 001

status in development

bull Initial commit

13

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 17: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

14 Chapter 2 Change history

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 18: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

CHAPTER

THREE

API REFERENCE

Release 00

Date March 20 2012

31 cymeclient

bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting

cymeclient

bull Python client for the Cyme HTTP API

311 Branches

gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]

gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000

312 Applications

gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo

15

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 19: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

313 Queues

gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)

gtgtgt appqueues[ltQueue ursquomyqueuersquogt]

314 Instances

gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo

gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo

gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]

gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1

gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10

gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80

rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo

rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True

rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0

315 Consumers

gtgtgt instanceconsumersadd(my_queue)ok ok

16 Chapter 3 API Reference

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 20: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

gtgtgt instance_consumersdelete(my_queue)ok ok

gtgtgt instanceconsumers consumers with full declarations

316 Deleting

This will delete the queue and eventually force all worker instances to stop consuming from it

gtgtgt my_queuedelete()

This will shutdown and delete an instance

gtgtgt idelete()

class cymeclientClient(url=None app=None info=None)

class Instances(client)

class Model(args kwargs)

class Consumers(client name)

create_model(data args kwargs)

class ClientInstancesModelLazyQueues(instance)

ClientInstancesModelautoscale(max=None min=None)

ClientInstancesModelqueues

ClientInstancesModelstats()

ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)

ClientInstancesautoscale(name max=None min=None)

ClientInstancescreate_model(data args kwargs)

ClientInstancesstats(name)

class ClientQueues(client)

class Model(parent args kwargs)

ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)

Clientadd(name broker=None arguments=None extra_config=None nowait=False)

Clientall()

Clientapp = lsquocymersquo

Clientbranch_info(id)

Clientbranches

31 cymeclient 17

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 21: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

Clientbuild_url(path)

Clientclone(app=None info=None)

Clientcreate_model(name info)

Clientdelete(name=None)

Clientget(name=None)

class cymeclientInstance(parent args kwargs)

argumentsA unicode string field

brokerA unicode string field

extra_configA unicode string field

is_enabledA boolean field type

max_concurrencyA field that validates input as an Integer

min_concurrencyA field that validates input as an Integer

nameA unicode string field

poolA unicode string field

queue_names

class cymeclientQueue(parent args kwargs)

exchangeA unicode string field

exchange_typeA unicode string field

nameA unicode string field

optionsA unicode string field

routing_keyA unicode string field

32 cymeclientbase

cymeclientbase

class cymeclientbaseBase

18 Chapter 3 API Reference

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 22: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

deserialize(text)

keys()

maybe_async(name nowait)

serialize(obj)

class cymeclientbaseClient(url=None)

DELETE(path params=None data=None type=None)

GET(path params=None type=None)

POST(path params=None data=None type=None)

PUT(path params=None data=None type=None)

default_url = lsquohttp1270018000rsquo

headers

request(method path params=None data=None type=None)

root(method path=None params=None data=None)

class cymeclientbaseModel(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

class cymeclientbasePath(s=None stack=None)

class cymeclientbaseSection(client)

class Model(parent args kwargs)

delete(nowait=False)

idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs

Sectionadd(name nowait=False data)

Sectionall()

Sectionall_names()

Sectioncreate_model(args kwargs)

Sectiondelete(name nowait=False)

Sectionget(name)

Sectionmaybe_async(name nowait)

Sectionname = None

Sectionpath = None

Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]

32 cymeclientbase 19

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 23: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

33 cymebranch

cymebranch

bull This is the Branch thread started by the cyme-branch program

It starts the HTTP server the Supervisor and one or more controllers

class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)

about()

after()

controller_cls = lsquocontrollerControllerrsquo

httpd_cls = lsquohttpdHttpServerrsquo

intsup_cls = lsquointsupgSuprsquo

on_ready(kwargs)

prepare_signals()

run()

stop()

supervisor_cls = lsquosupervisorSupervisorrsquo

class cymebranchMockSup(thread args)

start()

stop()

34 cymebranchcontroller

35 cymebranchmanagers

cymebranchmanagers

bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them

class cymebranchmanagersLocalInstanceManager

Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt

Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)

add_consumer(name queue nowait=False)

cancel_consumer(name queue nowait=False)

disable(name nowait=False)

20 Chapter 3 API Reference

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 24: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

enable(name nowait=False)

get(name)

maybe_wait(fun instances nowait)

remove(name nowait=False)

remove_queue(queue nowait=False)

restart(name nowait=False)

36 cymebranchsupervisor

cymebranchsupervisor

class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync

Parameters

bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances

bull queue ndash Custom Queue instance used to send and receive commands

It is responsible for

bullStopping removed instances

bullStarting new instances

bullRestarting unresponsivekilled instances

bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies

bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies

The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker

Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)

before()

interval = 600Default interval (time in seconds as a float to reschedule)

pause()Pause all timers

paused = FalseConnection errors pauses the supervisor so events does not accumulate

restart(instances)Restart one or more instances

Parameters instances ndash List of instances to restart

36 cymebranchsupervisor 21

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 25: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled

resume()Resume all timers

run()

shutdown(instances)Shutdown one or more instances

Parameters instances ndash List of instances to stop

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled

verify(instances ratelimit=False)Verify the consistency of one or more instances

Parameters instances ndash List of instances to verify

This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete

wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs

cymebranchsupervisorget_current()

cymebranchsupervisorset_current(sup)

37 cymebranchhttpd

cymebranchhttpd

bull Our embedded WSGI server used to serve the HTTP API

class cymebranchhttpdHttpServer(addrport=None)

create_http_protocol()

create_log()

joinable = False

logger_name

run()

server(sock handler)

url

22 Chapter 3 API Reference

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 26: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

38 cymebranchsignals

cymebranchsignals

cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve

Arguments

sender is the Branch instance

cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready

Arguments

sender is the Controller instance

cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments

sender the HttpServer instance

addrport the (hostname port) tuple

handler the WSGI handler used

sock the socket used

cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments

sender is the Supervisor instance

39 cymebranchstate

cymebranchstate

bull Global branch state

bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky

class cymebranchstateState

broker_last_revived = None

is_branch = Falseset to true if the process is a cyme-branch

on_broker_revive(args kwargs)

supervisor

time_since_broker_revived

310 cymebranchmetrics

cymebranchmetrics

38 cymebranchsignals 23

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 27: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

class cymebranchmetricsdf(path)

available

capacity

stat

total_blocks

cymebranchmetricsload_average()

311 cymebranchthread

cymebranchthread

bull Utilities for working with greenlets

exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started

class cymebranchthreadgThread

AlreadyStartedalias of AlreadyStartedError

exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds

When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly

Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it

cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application

pendingTrue if the timeout is scheduled to be raised

start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled

gThreadafter()Call after the thread has shut down

gThreadbefore()Called at the beginning of start()

gThreadextra_shutdown_steps = 0

gThreadextra_startup_steps = 0

gThreadg = NoneGreenlet instance of the thread set when the thread is started

24 Chapter 3 API Reference

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 28: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

gThreadjoin(timeout=None)Wait until the thread exits

Parameters timeout ndash Timeout in seconds (intfloat)

Raises eventletTimeout if the thread canrsquot be joined before the provided timeout

gThreadjoinable = TrueSet this to False if it is not possible to join the thread

gThreadkill()Kill the green thread

gThreadlogger_name

gThreadname = NoneName of the thread used in logs and such

gThreadping(timeout=None)

gThreadrespond_to_ping()

gThreadrun()

gThreadshould_stop = FalseSet when the thread is requested to stop

gThreadspawn(fun args kwargs)

gThreadstart()Spawn green thread and returns GreenThread instance

gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds

Parameters

bull interval ndash Interval in seconds (intfloat)

bull fun ndash The function to apply

bull args ndash Additional arguments to pass

bull kwargs ndash Additional keyword arguments to pass

Returns entry object with cancel and kill methods

gThreadstop(join=True timeout=1e+100)Shutdown the thread

This will also cancel+kill any periodic timers registered by the thread

Parameters

bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())

bull timeout ndash Timeout for join (default is 1e+100)

312 cymebranchintsup

cymebranchintsup

bull Internal supervisor used to ensure our threads are still running

312 cymebranchintsup 25

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 29: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

class cymebranchintsupgSup(thread signal interval=5 timeout=600)

logger_name

run()

start_wait_child()

stop()

313 cymeapiviews

314 cymeapiweb

cymeapiweb

bull Contains utilities for creating our HTTP API

class cymeapiwebApiView(kwargs)

Accepted(args kwargs)

Created(data args kwargs)

NotImplemented(args kwargs)

Ok(data args kwargs)

Response(args kwargs)

dispatch(request args kwargs)

get_or_post(key default=None)

get_param(key type=None)

nowait = False

params(keys)

typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt

class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)

The requested action is not implemented Used for async requests when the operation is inherently sync

status_code = 501

class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)

The operation timed out

status_code = 408

cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response

cymeapiwebset_access_control_options(response options=None)

cymeapiwebsimple_get(fun)

26 Chapter 3 API Reference

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 30: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

315 cymemodels

cymemodels

class cymemodelsBroker(args kwargs)Broker connection arguments

urlAMQP (kombu) url

connection()Return a new Connection to this broker

as_dict()Returns this broker as a dictionary that can be Json encoded

poolA connection pool with connections to this broker

objectsThe manager for this model is BrokerManager

class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances

nameQueue name (unique max length 128)

exchangeExchange name (max length 128)

exchange_typeExchange type (max length 128)

routing_keyRouting keybinding key (max length 128)

optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported

is_enabledNot in use

created_atTimestamp created

as_dict()Returns dictionary representation of this queue that can be Json encoded

objectsThe manager for this model is QueueManager

class cymemodelsInstance(args kwargs)A celeryd instance

nameName of the instance

queuesQueues this instance should consume from (many to many relation to Queue)

315 cymemodels 27

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 31: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)

min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)

is_enabledFlag set if this instance should be running

created_atTimestamp of when this instance was first created

brokerThe broker this instance should connect to (foreign key to Broker)

BrokerBroker model class used (default is Broker)

QueueQueue model class used (default is Queue)

MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)

objectsThe manager used for this model is InstanceManager

cwdWorking directory used by all instances (Default is varruncyme)

as_dict()Returns dictionary representation of this instance that can be Json encoded

enable()Enables this instance (model-only)

disable()Disables this instance (model-only)

start(kwargs)Starts the instance

stop(kwargs)Shuts down the instance

restart(kwargs)Restarts the instance

alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts

stats(kwargs)Returns instance statistics (like celeryctl inspect stats)

autoscale(max=None min=None kwargs)Set maxmin autoscale settings

responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping

28 Chapter 3 API Reference

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 32: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

responds_to_signal()Returns True if the pid file exists and the pid responds to signals

consuming_from(kwargs)Returns the queues the instance is currently consuming from

add_queue(q kwargs)Add queue for this instance by name

cancel_queue(queue kwargs)Cancel queue for this instance by Queue

getpid()Get the process id for this instance by reading its pid file

Returns None if the pid file does not exist

_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command

_query(cmd args= kwargs)Send remote control command and wait for this instances reply

316 cymemodelsmanagers

cymemanagers

bull These are the managers for our models in cymemodels

bull They are not to be used directly but accessed through the objects attribute of a Model

class cymemodelsmanagersAppManager

Brokers

add(name=None broker=None arguments=None extra_config=None)

from_json(name=None broker=None)

get_broker(url)

get_default()

instance(name=None broker=None)

recreate(name=None broker=None arguments=None extra_config=None)

class cymemodelsmanagersBrokerManager

default_url

get_default()

class cymemodelsmanagersInstanceManager

add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)

add_queue_to_instances(queue query)

disable(name)

316 cymemodelsmanagers 29

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 33: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

enable(name)

enabled()

remove(name)

remove_queue_from_instances(queue query)

class cymemodelsmanagersQueueManager

add(name exchange=None exchange_type=None routing_key=None options)

enabled()

317 cymestatus

cymestatus

class cymestatusStatus

all_instances()

ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))

insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures

pause()

paused = False

respond_to_ping()

restart_all()

restart_max_rate = lsquo100srsquo

resume()

shutdown_all()

start_all()

318 cymetasks

cymetasks

bull We have our own version of the webhook task

bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response

cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request

cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string

cymetasksresponse_to_dict(r)

30 Chapter 3 API Reference

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 34: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

319 cymemanagementcommandscyme

bull cymendash Options

3191 cyme

Cyme management utility

Options

-a -appApplication to use Required for all operations except for when creating deleting or listing apps

-n -nowaitDonrsquot want for operations to complete (async)

-F -formatOutput format pretty (default) or json

-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom logfile path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user

This needs to be properly specified if the -L option is used

-b -brokerBroker to use for a local branches request

class cymemanagementcommandscymeCommand(env=None args kwargs)

args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo

create_superuser()

drop_into_shell()

handle(args kwargs)

help = lsquoCyme management utilityrsquo

name = lsquocymersquo

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)

restart_all()

shutdown_all()

319 cymemanagementcommandscyme 31

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 35: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

start_all()

status

class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)

DISPATCH(fqdn args)

add_app(args kwargs)

add_consumer(args kwargs)

add_instance(args kwargs)

add_queue(args kwargs)

all_apps(args kwargs)

all_consumers(args kwargs)

all_instances(args kwargs)

all_queues(args kwargs)

delete_app(args kwargs)

delete_consumer(args kwargs)

delete_instance(args kwargs)

delete_queue(args kwargs)

format_optargs(optargs)

format_response(ret)

get_app(args kwargs)

get_instances(args kwargs)

get_queue(args kwargs)

getsig(fun opt_args=None)

instance_autoscale(args kwargs)

instance_stats(args kwargs)

prepare_response(ret)

class cymemanagementcommandscymeLocalI(args kwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

32 Chapter 3 API Reference

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 36: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app()

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)

add_app(name broker=None arguments=None extra_config=None)

add_consumer(instance_name queue_name)

add_instance(name=None broker=None arguments=None extra_config=None)

add_queue(name exchange=None exchange_type=None routing_key=None options=None)

all_apps()

all_branches()

all_consumers(instance_name)

all_instances()

all_queues()

client

delete_app(name)

delete_consumer(instance_name queue_name)

delete_instance(name)

delete_queue(name)

get_app(name)

get_instance(name)

get_queue(name)

instance_autoscale(name max=None min=None)

instance_stats(name)

prepare_response(ret)

320 cymemanagementcommandscyme_branch

bull cyme-branchndash Options

320 cymemanagementcommandscyme_branch 33

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 37: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

3201 cyme-branch

Starts the cyme branch service

Options

-i -idSet branch id if not provided one will be automatically generated

-without-httpdDisable the HTTP server thread

-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO

-f -logfileSet custom log file path Default is ltstderrgt

-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as

-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2

-sup-intervalSupervisor schedule Interval in seconds Default is 5

class cymemanagementcommandscyme_branchCommand(env=None args kwargs)

args = lsquo[optional port number or ipaddrport]rsquo

banner()

branch_cls = lsquocymebranchBranchrsquo

connect_signals()

default_detach_logfile = lsquob ranchlogrsquo

default_detach_pidfile = lsquobranchpidrsquo

handle(args kwargs)

help = lsquoStarts a cyme branchrsquo

install_signal_handlers()

name = lsquocyme-branchrsquo

on_branch_ready(sender=None kwargs)

on_branch_shutdown(sender=None kwargs)

option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)

repr_controller_id(c)

set_process_title(info)

setup_default_env(env)

setup_shutdown_progress(sender=None kwargs)

setup_startup_progress(sender=None kwargs)

34 Chapter 3 API Reference

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 38: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

signals

stop()

321 cymebinbase

cymebinbase

bull Utilities used by command line applications basically just to set up Django without having an actual project

class cymebinbaseBaseApp

env = None

get_version()

instance_dir = None

needs_eventlet = False

run_from_argv(argv=None)

class cymebinbaseEnv(needs_eventlet=False instance_dir=None)

configure()

management

setup_eventlet()

setup_pool_limit(kwargs)

syncdb(interactive=True)

cymebinbaseapp(attrs)

322 cymebincyme

cymebincyme

323 cymebincyme_branch

cymebincyme_branch

bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)

bull It in turn executes the cyme-branch management command

324 cymeutils

cymeutils

321 cymebinbase 35

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 39: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

class cymeutilsLazyProgressBar(size description=None endtext=None)

finish(kwargs)

step(i=1 kwargs)

class cymeutilsPathunipathPath version that can use the operator to combine paths

gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)

cymeutilsfind_package(mod _s=None)Find the package a module belongs to

if you have structure

package__init__pyfoopybar__init__pybarbazpy

Then the following examples returns

gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar

Note that this does not look at the file system but rather uses the __package__ attribute of a module

cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin

Eg if origin is an object in the module packagefoo then

gtgtgt find_symbol(origin barmy_symbol)

will return the object my_symbol from module packagebar

cymeutilsforce_list(obj)Force object to be a list

If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list

cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence

cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs

cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)

See celerylogLogredirect_stdouts_to_logger()

cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile

stderr will be used if not logfile provided

36 Chapter 3 API Reference

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 40: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

CHAPTER

FOUR

INDICES AND TABLES

bull genindex

bull modindex

bull search

37

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 41: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

Cyme Documentation Release 004

38 Chapter 4 Indices and tables

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39

Page 42: Cyme Documentation - Read the Docs · Cyme Documentation, Release 0.0.4 exchange and routing_key will default to the queue name if not provided, and exchange_type will default to

PYTHON MODULE INDEX

ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch

cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils

39


Recommended