Cyme DocumentationRelease 004
VMWare Inc
March 20 2012
CONTENTS
i
ii
Cyme Documentation Release 004
Contents
CONTENTS 1
Cyme Documentation Release 004
2 CONTENTS
CHAPTER
ONE
INTRODUCTION
bull Synopsisbull Requirementsbull Getting startedbull API Reference
ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale
bull Componentsndash Programsndash Models
App Broker Instance Queue
bull Supervisorbull Controllerbull HTTP
11 Synopsis
Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service
12 Requirements
bull Python 26 or later
bull cl
bull eventlet
3
Cyme Documentation Release 004
bull django
bull django-celery
Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios
13 Getting started
You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch
Start one or more branches
$ cyme-branch 8001 -i branch1 -D cymebranch1
$ cyme-branch 8002 -i branch2 -D cymebranch2
The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory
Create a new application named foo
$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked
name foo broker password guesthostname 127001userid guestport 5672virtual_host
Note that we can edit the broker connection details here by using a POST request
$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo
Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed
Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data
Create a new Celery worker instance
$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked
is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712
4 Chapter 1 Introduction
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
CONTENTS
i
ii
Cyme Documentation Release 004
Contents
CONTENTS 1
Cyme Documentation Release 004
2 CONTENTS
CHAPTER
ONE
INTRODUCTION
bull Synopsisbull Requirementsbull Getting startedbull API Reference
ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale
bull Componentsndash Programsndash Models
App Broker Instance Queue
bull Supervisorbull Controllerbull HTTP
11 Synopsis
Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service
12 Requirements
bull Python 26 or later
bull cl
bull eventlet
3
Cyme Documentation Release 004
bull django
bull django-celery
Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios
13 Getting started
You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch
Start one or more branches
$ cyme-branch 8001 -i branch1 -D cymebranch1
$ cyme-branch 8002 -i branch2 -D cymebranch2
The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory
Create a new application named foo
$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked
name foo broker password guesthostname 127001userid guestport 5672virtual_host
Note that we can edit the broker connection details here by using a POST request
$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo
Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed
Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data
Create a new Celery worker instance
$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked
is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712
4 Chapter 1 Introduction
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
ii
Cyme Documentation Release 004
Contents
CONTENTS 1
Cyme Documentation Release 004
2 CONTENTS
CHAPTER
ONE
INTRODUCTION
bull Synopsisbull Requirementsbull Getting startedbull API Reference
ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale
bull Componentsndash Programsndash Models
App Broker Instance Queue
bull Supervisorbull Controllerbull HTTP
11 Synopsis
Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service
12 Requirements
bull Python 26 or later
bull cl
bull eventlet
3
Cyme Documentation Release 004
bull django
bull django-celery
Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios
13 Getting started
You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch
Start one or more branches
$ cyme-branch 8001 -i branch1 -D cymebranch1
$ cyme-branch 8002 -i branch2 -D cymebranch2
The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory
Create a new application named foo
$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked
name foo broker password guesthostname 127001userid guestport 5672virtual_host
Note that we can edit the broker connection details here by using a POST request
$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo
Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed
Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data
Create a new Celery worker instance
$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked
is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712
4 Chapter 1 Introduction
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
Contents
CONTENTS 1
Cyme Documentation Release 004
2 CONTENTS
CHAPTER
ONE
INTRODUCTION
bull Synopsisbull Requirementsbull Getting startedbull API Reference
ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale
bull Componentsndash Programsndash Models
App Broker Instance Queue
bull Supervisorbull Controllerbull HTTP
11 Synopsis
Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service
12 Requirements
bull Python 26 or later
bull cl
bull eventlet
3
Cyme Documentation Release 004
bull django
bull django-celery
Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios
13 Getting started
You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch
Start one or more branches
$ cyme-branch 8001 -i branch1 -D cymebranch1
$ cyme-branch 8002 -i branch2 -D cymebranch2
The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory
Create a new application named foo
$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked
name foo broker password guesthostname 127001userid guestport 5672virtual_host
Note that we can edit the broker connection details here by using a POST request
$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo
Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed
Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data
Create a new Celery worker instance
$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked
is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712
4 Chapter 1 Introduction
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
2 CONTENTS
CHAPTER
ONE
INTRODUCTION
bull Synopsisbull Requirementsbull Getting startedbull API Reference
ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale
bull Componentsndash Programsndash Models
App Broker Instance Queue
bull Supervisorbull Controllerbull HTTP
11 Synopsis
Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service
12 Requirements
bull Python 26 or later
bull cl
bull eventlet
3
Cyme Documentation Release 004
bull django
bull django-celery
Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios
13 Getting started
You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch
Start one or more branches
$ cyme-branch 8001 -i branch1 -D cymebranch1
$ cyme-branch 8002 -i branch2 -D cymebranch2
The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory
Create a new application named foo
$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked
name foo broker password guesthostname 127001userid guestport 5672virtual_host
Note that we can edit the broker connection details here by using a POST request
$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo
Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed
Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data
Create a new Celery worker instance
$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked
is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712
4 Chapter 1 Introduction
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
CHAPTER
ONE
INTRODUCTION
bull Synopsisbull Requirementsbull Getting startedbull API Reference
ndash Applicationsndash Instancesndash Queuesndash Consumersndash Queueing Tasksndash Querying Task Statendash Instance details and statisticsndash Autoscale
bull Componentsndash Programsndash Models
App Broker Instance Queue
bull Supervisorbull Controllerbull HTTP
11 Synopsis
Cyme is the Celery instance manager a distributed application that manages clusters of Celery worker instances Everymachine (physical or virtual) runs the cyme-branch service
12 Requirements
bull Python 26 or later
bull cl
bull eventlet
3
Cyme Documentation Release 004
bull django
bull django-celery
Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios
13 Getting started
You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch
Start one or more branches
$ cyme-branch 8001 -i branch1 -D cymebranch1
$ cyme-branch 8002 -i branch2 -D cymebranch2
The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory
Create a new application named foo
$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked
name foo broker password guesthostname 127001userid guestport 5672virtual_host
Note that we can edit the broker connection details here by using a POST request
$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo
Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed
Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data
Create a new Celery worker instance
$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked
is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712
4 Chapter 1 Introduction
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
bull django
bull django-celery
Note An updated list of requirements can always be found in the requirements directory of the Cyme distri-bution This directory contains pip requirements files for different scenarios
13 Getting started
You can run as many branches as needed one or multiple It is also possible to run multiple branches on the samemachine by specifying a custom HTTP port and root directory for each branch
Start one or more branches
$ cyme-branch 8001 -i branch1 -D cymebranch1
$ cyme-branch 8002 -i branch2 -D cymebranch2
The default HTTP port is 8000 and the default root directory is instances The root directory must be writableby the user the cyme-branch application is running as The logs and pid files of every worker instance will bestored in this directory
Create a new application named foo
$ curl -X POST -i httplocalhost8001fooHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 220643 GMTTransfer-Encoding chunked
name foo broker password guesthostname 127001userid guestport 5672virtual_host
Note that we can edit the broker connection details here by using a POST request
$ curl -X POST -i httplocalhostbar -d rsquohostname=w1ampuserid=meamppassword=meampvhost=rsquo
Note For convenience and full client support PUT can be replaced with a POST instead and it will result in thesame action being performed
Also for POST PUT and DELETE the query part of the URL can be used instead of actual post data
Create a new Celery worker instance
$ curl -X PUT -i httplocalhost8001fooinstancesHTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 152511 GMTTransfer-Encoding chunked
is_enabled truename a35f2518-13bb-4403-bbdf-dd8751077712
4 Chapter 1 Introduction
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
queues []broker password guest
userid guesthostname 127001virtual_host port 5672
max_concurrency 1min_concurrency 1
Note that this instance is created on a random branch not necessarily the branch that you are currently speaking toover HTTP If you want to edit the data on a specific branch please do so by using the admin interface of the branchat httplocalhost8001admin
In the logs of the affected branch you should now see something like this
582161d7-1187-4242-9874-32cd7186ba91 --gt Instanceadd(name=None)Supervisor wake-upSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 instancerestartceleryd-multi restart --suffix= --no-color a35f2518-13bb-4403-bbdf-dd8751077712
-Q rsquodqa35f2518-13bb-4403-bbdf-dd8751077712rsquo--workdir=cymebranch1--pidfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerpid--logfile=cymebranch1a35f2518-13bb-4403-bbdf-dd8751077712workerlog--loglevel=DEBUG --autoscale=11--broker=amqpguestguestlocalhost5672
celeryd-multi v231gt a35f2518-13bb-4403-bbdf-dd8751077712 DOWNgt Restarting instance a35f2518-13bb-4403-bbdf-dd8751077712 OKSupervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 01Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 05Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 pingWithTimeout 09Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 successfully restartedSupervisor wake-up582161d7-1187-4242-9874-32cd7186ba91 lt-- ok=
rsquois_enabledrsquo Truersquonamersquo rsquoa35f2518-13bb-4403-bbdf-dd8751077712rsquorsquoqueuesrsquo []rsquobrokerrsquo rsquopasswordrsquo ursquoguestrsquo
rsquohostnamersquo ursquo127001rsquorsquouseridrsquo ursquoguestrsquorsquoportrsquo 5672rsquovirtual_hostrsquo ursquorsquo
rsquomax_concurrencyrsquo 1rsquomin_concurrencyrsquo 1
Now that we have created an instance we can list the available instances
$ curl -X GET -i httplocalhost8001fooinstancesHTTP11 200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 152833 GMTTransfer-Encoding chunked
[a35f2518-13bb-4403-bbdf-dd8751077712]
Note that this will list instances for every branch not just the branch you are currently speaking to over HTTP
Letrsquos create a queue declaration for a queue named tasks This queue binds the exchange tasks with routing keytasks (note that the queue name will be used as both exchange name and routing key if these are not provided)
13 Getting started 5
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
Create the queue by performing the following request
$ curl -X POST -d rsquoexchange=tasksamprouting_key=tasksrsquo -i httplocalhost8001fooqueuestasks
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160307 GMTTransfer-Encoding chunked
exchange t2routing_key t2options nullname t2exchange_type null
The queue declaration should now have been stored inside one of the branches and we can verify that by retrieving alist of all queues defined on all branches
$ curl -X GET -i httplocalhost8001fooqueuesHTTP11200 OKContent-Type applicationjsonDate Mon 15 Aug 2011 160837 GMTTransfer-Encoding chunked
[tasks]
Now we can make our worker instance consume from the tasks queue to process tasks sent to it
$ curl -X PUT -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd8751077712queuest2
HTTP11 201 CREATEDContent-Type applicationjsonDate Mon 15 Aug 2011 160632 GMTTransfer-Encoding chunked
ok ok
In the logs for the branch that this is instance is a member of you should now see
[2011-08-15 160632226 WARNINGMainProcess]Supervisor a35f2518-13bb-4403-bbdf-dd8751077712 instanceconsume_from tasks
If the test was successful you can clean up after yourself by
bull Cancelling consuming from the tasks queue
$ curl -X DELETE -i httplocalhost8001fooinstancesa35f2518-13bb-4403-bbdf-dd875107772queuestasks
bull Deleting the tasks queue
$ curl -X DELETE -i httplocalhost8001fooqueues
bull and finally deleting the worker instance
$ curl -X DELETE -i httplocalhost8001instancesa35f2518-13bb-4403-bbdf-dd8751077712
The worker instance should now be shutdown by the branch supervisor
6 Chapter 1 Introduction
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
14 API Reference
141 Applications
bull Create new named application
[PUT|POST] httpbranchportltnamegthostname=strport=intuserid=strpassword=strvirtual_host=str
If hostname is not provided then any other broker parameters will be ignored and the default broker will be used
bull List all available applications
GET httpbranchport
bull Get the configuration for app by name
GET httpbranchportname
142 Instances
bull Create and start an anonymous instance associated with app
[PUT|POST] httpbranchportltappgtinstances
This will return the details of the new id including the instance name (which for anonymous instances is an UUID)
bull Create and start a named instance associated with app
[PUT|POST] httpbranchportltappgtinstancesltnamegt
bull List all available instances associated with an app
GET httpbranchportltappgt
bull Get the details of an instance by name
GET httpbranchportltappgtinstancesltnamegt
bull Delete an instance by name
DELETE httpbranchportltappgtinstancesltnamegt
143 Queues
bull Create a new queue declaration by name
[PUT|POST] httpbranchportltappgtqueuesltnamegtexchange=strexchange_type=strrouting_key=stroptions=json dict
14 API Reference 7
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
exchange and routing_key will default to the queue name if not provided and exchange_type will defaultto direct options is a json encoded mapping of additional queue exchange and binding options for a full list ofsupported options see kombucompatentry_to_queue()
bull Get the declaration for a queue by name
GET httpbranchportltappgtqueuesltnamegt
bull Get a list of available queues
GET httpbranchportltappgtqueues
144 Consumers
Every instance can consume from one or more queues Queues are referred to by name and there must exist a fulldeclaration for that name
bull Tell an instance by name to consume from queue by name
[PUT|POST] httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
bull Tell an instance by name to stop consuming from queue by name
DELETE httpbranchportltappgtinstancesltinstancegtqueuesltqueuegt
145 Queueing Tasks
Queueing an URL will result in one of the worker instances to execute that request as soon as possible
[verb] httpbranchportltappgtqueueltqueuegtlturlgtget_datapost_data
The verb can be any supported HTTP verb such as HEAD GET POST PUT DELETE TRACE OPTIONSCONNECT and PATCH The worker will then use the same verb when performing the request Any get and postdata provided will also be forwarded
When you queue an URL a unique identifier is returned you can use this identifier (called an UUID) to query thestatus of the task or collect the return value The return value of the task is the HTTP response of the actual requestperformed by the worker
Examples
GET httpbranchportltappgtqueuetaskshttpmimport_contactsuser=133
POST httpbranchportltappgtqueuetaskshttpmimport_userusername=George Costanzacompany=Vandelay Industries
146 Querying Task State
bull To get the current state of a task
GET httpbranchportltappgtqueryltuuidgtstate
bull To get the return value of a task
8 Chapter 1 Introduction
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
GET httpbranchportltappgtqueryltuuidgtresult
bull To wait for a task to complete and return its result
GET httpbranchportltappgtqueryltuuidgtwait
147 Instance details and statistics
To get configuration details and statistics for a particular instance
GET httpbranchportltappgtinstanceltnamegtstats
148 Autoscale
bull To set the maxmin concurrency settings of an instance
POST httpbranchportltappgtinstanceltnamegtautoscalemax=intmin=int
bull To get the maxmin concurrency settings of an instance
GET httpbranchportltappgtinstanceltnamegtautoscale
15 Components
151 Programs
bull cyme ltcymemanagementcommandscyme
This is the management application speaking HTTP with the clients See cyme --help for fulldescription and command line arguments
bull cyme-branch
Creates a new branch and starts the service to manage it See cyme-branch --help for fulldescription and command line arguments
152 Models
The branch manager uses an SQLite database to store state but this can also be another database system (MySQLPostgreSQL Oracle DB2)
App
see cymemodelsApp
Every instance belongs to an application and the application contains the default broker configuration
15 Components 9
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
Broker
see cymemodelsBroker
The connection parameters for a specific broker (hostname port userid password virtual_host)
Instance
see cymemodelsInstance
This describes a Celery worker instance that is a member of this branch And also the queues it should consume fromand its maxmin concurrency settings It also describes what broker the instance should be connecting to (which if notspecified will default to the broker of the app the instance belongs to)
Queue
see cymemodelsQueue
A queue declaration name exchange exchange type routing key and options Options is a json encoded mapping ofqueue exchange and binding options supported by kombucompatentry_to_queue()
16 Supervisor
see cymesupervisor
The supervisor wakes up at intervals to monitor for changes in the model It can also be requested to perform specificoperations eg restart an instance add queues to instance and these operations can be either async or sync
It is responsible for
bull Stopping removed instances
bull Starting new instances
bull Restarting unresponsivekilled instances
bull Making sure the instances consumes from the queues specified in the model sending add_consumer- can-cel_consumer broadcast commands to the instances as it finds inconsistencies
bull Making sure the maxmin concurrency setting is as specified in the model sending autoscale broadcast com-mands to the instances as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependent on abroker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisor will not restartaffected instances until the instance has had a chance to reconnect (decided by the wait_after_broker_revived attribute)
17 Controller
see cymecontroller
The controller is a series of cl actors to control applications instances and queues It is used by the HTTP interfacebut can also be used directly
10 Chapter 1 Introduction
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
18 HTTP
The http server currently serves up an admin instance where you can add remove and modify instances
The http server can be disabled using the --without-http option
18 HTTP 11
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
12 Chapter 1 Introduction
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
CHAPTER
TWO
CHANGE HISTORY
bull 001
21 001
status in development
bull Initial commit
13
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
14 Chapter 2 Change history
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
CHAPTER
THREE
API REFERENCE
Release 00
Date March 20 2012
31 cymeclient
bull Branchesbull Applicationsbull Queuesbull Instancesbull Consumersbull Deleting
cymeclient
bull Python client for the Cyme HTTP API
311 Branches
gtgtgt client = Client(httplocalhost8000)gtgtgt clientbranches[cyme1examplecom cyme2examplecom]
gtgtgt clientbranch_info(cyme1examplecom)rsquosup_intervalrsquo 5 rsquonumcrsquo 2 rsquologlevelrsquo rsquoINFOrsquorsquologfilersquo None rsquoidrsquo rsquocyme1examplecomrsquo rsquoportrsquo 8000
312 Applications
gtgtgt app = clientget(foo)gtgtgt appltClient rsquohttplocalhost8016foorsquogtgtgtgt appinforsquonamersquo rsquofoorsquo rsquobrokerrsquo rsquoamqpguestguestlocalhost5672rsquo
15
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
313 Queues
gtgtgt appqueuesadd(myqueue exchange=myex routing_key=x)gtgtgt ltQueue ursquomyqueuersquogtgtgtgt my_queue = appqueuesget(my_queue)
gtgtgt appqueues[ltQueue ursquomyqueuersquogt]
314 Instances
gtgtgt i = appinstancesadd()gtgtgt iltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogtgtgtgt inameursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquo
gtgtgt ibroker lt inherited from appursquoamqpguestguestlocalhost5672rsquo
gtgtgt appinstances[ltInstance ursquod87798f3-0bb0-4161-8e0b-a5f069b1d58brsquogt]
gtgtgt iautoscale() current autoscale settingsrsquomaxrsquo 1 rsquominrsquo 1
gtgtgt iautoscale(max=10 min=10) always run 10 processesrsquomaxrsquo 10 rsquominrsquo 10
gtgtgt istats()rsquototalrsquo rsquoconsumerrsquo rsquoprefetch_countrsquo 80
rsquobrokerrsquo rsquotransport_optionsrsquo rsquologin_methodrsquo rsquoAMQPLAINrsquorsquohostnamersquo rsquo127001rsquorsquouseridrsquo rsquoguestrsquorsquoinsistrsquo Falsersquoconnect_timeoutrsquo 4rsquosslrsquo Falsersquovirtual_hostrsquo rsquorsquorsquoportrsquo 5672rsquotransportrsquo rsquoamqprsquo
rsquopoolrsquo rsquotimeoutsrsquo [None None]rsquoprocessesrsquo [76003]rsquomax-concurrencyrsquo 1rsquomax-tasks-per-childrsquo Nonersquoput-guarded-by-semaphorersquo True
rsquoautoscalerrsquo rsquocurrentrsquo 1 rsquomaxrsquo 1 rsquominrsquo 1 rsquoqtyrsquo 0
315 Consumers
gtgtgt instanceconsumersadd(my_queue)ok ok
16 Chapter 3 API Reference
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
gtgtgt instance_consumersdelete(my_queue)ok ok
gtgtgt instanceconsumers consumers with full declarations
316 Deleting
This will delete the queue and eventually force all worker instances to stop consuming from it
gtgtgt my_queuedelete()
This will shutdown and delete an instance
gtgtgt idelete()
class cymeclientClient(url=None app=None info=None)
class Instances(client)
class Model(args kwargs)
class Consumers(client name)
create_model(data args kwargs)
class ClientInstancesModelLazyQueues(instance)
ClientInstancesModelautoscale(max=None min=None)
ClientInstancesModelqueues
ClientInstancesModelstats()
ClientInstancesadd(name=None broker=None arguments=None config=Nonenowait=False)
ClientInstancesautoscale(name max=None min=None)
ClientInstancescreate_model(data args kwargs)
ClientInstancesstats(name)
class ClientQueues(client)
class Model(parent args kwargs)
ClientQueuesadd(name exchange=None exchange_type=None routing_key=Nonenowait=False options)
Clientadd(name broker=None arguments=None extra_config=None nowait=False)
Clientall()
Clientapp = lsquocymersquo
Clientbranch_info(id)
Clientbranches
31 cymeclient 17
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
Clientbuild_url(path)
Clientclone(app=None info=None)
Clientcreate_model(name info)
Clientdelete(name=None)
Clientget(name=None)
class cymeclientInstance(parent args kwargs)
argumentsA unicode string field
brokerA unicode string field
extra_configA unicode string field
is_enabledA boolean field type
max_concurrencyA field that validates input as an Integer
min_concurrencyA field that validates input as an Integer
nameA unicode string field
poolA unicode string field
queue_names
class cymeclientQueue(parent args kwargs)
exchangeA unicode string field
exchange_typeA unicode string field
nameA unicode string field
optionsA unicode string field
routing_keyA unicode string field
32 cymeclientbase
cymeclientbase
class cymeclientbaseBase
18 Chapter 3 API Reference
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
deserialize(text)
keys()
maybe_async(name nowait)
serialize(obj)
class cymeclientbaseClient(url=None)
DELETE(path params=None data=None type=None)
GET(path params=None type=None)
POST(path params=None data=None type=None)
PUT(path params=None data=None type=None)
default_url = lsquohttp1270018000rsquo
headers
request(method path params=None data=None type=None)
root(method path=None params=None data=None)
class cymeclientbaseModel(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
class cymeclientbasePath(s=None stack=None)
class cymeclientbaseSection(client)
class Model(parent args kwargs)
delete(nowait=False)
idA field that stores a valid UUID value and optionally auto-populates empty values with new UUIDs
Sectionadd(name nowait=False data)
Sectionall()
Sectionall_names()
Sectioncreate_model(args kwargs)
Sectiondelete(name nowait=False)
Sectionget(name)
Sectionmaybe_async(name nowait)
Sectionname = None
Sectionpath = None
Sectionproxy = [rsquoGETrsquo lsquoPOSTrsquo lsquoPUTrsquo lsquoDELETErsquo]
32 cymeclientbase 19
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
33 cymebranch
cymebranch
bull This is the Branch thread started by the cyme-branch program
It starts the HTTP server the Supervisor and one or more controllers
class cymebranchBranch(addrport=rsquolsquo id=None loglevel=20 logfile=None without_httpd=Falsenumc=2 sup_interval=None ready_event=None colored=None kwargs)
about()
after()
controller_cls = lsquocontrollerControllerrsquo
httpd_cls = lsquohttpdHttpServerrsquo
intsup_cls = lsquointsupgSuprsquo
on_ready(kwargs)
prepare_signals()
run()
stop()
supervisor_cls = lsquosupervisorSupervisorrsquo
class cymebranchMockSup(thread args)
start()
stop()
34 cymebranchcontroller
35 cymebranchmanagers
cymebranchmanagers
bull Contains the LocalInstanceManager instance which is the preferred API used to control and manageworker instances handled by this branch Ie it can be used to do synchronous actions that donrsquot return until thesupervisor has performed them
class cymebranchmanagersLocalInstanceManager
Brokers = ltcymemodelsmanagersBrokerManager object at 0x4d08f90gt
Instances = ltcymemodelsmanagersInstanceManager object at 0x4d18290gt
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None nowait=False kwargs)
add_consumer(name queue nowait=False)
cancel_consumer(name queue nowait=False)
disable(name nowait=False)
20 Chapter 3 API Reference
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
enable(name nowait=False)
get(name)
maybe_wait(fun instances nowait)
remove(name nowait=False)
remove_queue(queue nowait=False)
restart(name nowait=False)
36 cymebranchsupervisor
cymebranchsupervisor
class cymebranchsupervisorSupervisor(interval=None queue=None set_as_current=True)The supervisor wakes up at intervals to monitor changes in the model It can also be requested to performspecific operations and these operations can be either async or sync
Parameters
bull interval ndash This is the interval (in seconds as an intfloat) between verifying all the registeredinstances
bull queue ndash Custom Queue instance used to send and receive commands
It is responsible for
bullStopping removed instances
bullStarting new instances
bullRestarting unresponsivekilled instances
bullMaking sure the instances consumes from the queues specified in the model sending add_consumer-cancel_consumer broadcast commands to the instances as it finds inconsistencies
bullMaking sure the maxmin concurrency setting is as specified in the model sending autoscale broadcastcommands to the noes as it finds inconsistencies
The supervisor is resilient to intermittent connection failures and will auto-retry any operation that is dependenton a broker
Since workers cannot respond to broadcast commands while the broker is off-line the supervisorwill not restart affected instances until the instance has had a chance to reconnect (decided by thewait_after_broker_revived attribute)
before()
interval = 600Default interval (time in seconds as a float to reschedule)
pause()Pause all timers
paused = FalseConnection errors pauses the supervisor so events does not accumulate
restart(instances)Restart one or more instances
Parameters instances ndash List of instances to restart
36 cymebranchsupervisor 21
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
restart_max_rate = lsquo1mrsquoLimit instance restarts to 1m so out of control instances will be disabled
resume()Resume all timers
run()
shutdown(instances)Shutdown one or more instances
Parameters instances ndash List of instances to stop
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
Warning Note that the supervisor will automatically restart any stopped instances unless the corre-sponding Instance model has been marked as disabled
verify(instances ratelimit=False)Verify the consistency of one or more instances
Parameters instances ndash List of instances to verify
This operation is asynchronous and returns a Greenlet instance that can be used to wait for the operationto complete
wait_after_broker_revived = 350Default interval_max for ensure_connection is 30 secs
cymebranchsupervisorget_current()
cymebranchsupervisorset_current(sup)
37 cymebranchhttpd
cymebranchhttpd
bull Our embedded WSGI server used to serve the HTTP API
class cymebranchhttpdHttpServer(addrport=None)
create_http_protocol()
create_log()
joinable = False
logger_name
run()
server(sock handler)
url
22 Chapter 3 API Reference
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
38 cymebranchsignals
cymebranchsignals
cymebranchsignalsbranch_ready = ltSignal SignalgtSent when the branch and all its components are ready to serve
Arguments
sender is the Branch instance
cymebranchsignalscontroller_ready = ltSignal SignalgtSent when a controller is ready
Arguments
sender is the Controller instance
cymebranchsignalshttpd_ready = ltSignal SignalgtSent when the http server is ready to accept requests Arguments
sender the HttpServer instance
addrport the (hostname port) tuple
handler the WSGI handler used
sock the socket used
cymebranchsignalssupervisor_ready = ltSignal SignalgtSent when the supervisor is ready Arguments
sender is the Supervisor instance
39 cymebranchstate
cymebranchstate
bull Global branch state
bull Used to keep track lost connections and so on which is used by the supervisor to know if an instance is actuallydown or if it is just the connection being shaky
class cymebranchstateState
broker_last_revived = None
is_branch = Falseset to true if the process is a cyme-branch
on_broker_revive(args kwargs)
supervisor
time_since_broker_revived
310 cymebranchmetrics
cymebranchmetrics
38 cymebranchsignals 23
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
class cymebranchmetricsdf(path)
available
capacity
stat
total_blocks
cymebranchmetricsload_average()
311 cymebranchthread
cymebranchthread
bull Utilities for working with greenlets
exception cymebranchthreadAlreadyStartedErrorRaised if trying to start a thread instance that is already started
class cymebranchthreadgThread
AlreadyStartedalias of AlreadyStartedError
exception Timeout(seconds=None exception=None)Raises exception in the current greenthread after timeout seconds
When exception is omitted or None the Timeout instance itself is raised If seconds is None the timeris not scheduled and is only useful if yoursquore planning to raise it directly
Timeout objects are context managers and so can be used in with statements When used in a withstatement if exception is False the timeout is still raised but the context manager suppresses it so thecode outside the with-block wonrsquot see it
cancel()If the timeout is pending cancel it If not using Timeouts in with statements always call cancel()in a finally after the block of code that is getting timed out If not canceled the timeout will beraised later on in some unexpected section of the application
pendingTrue if the timeout is scheduled to be raised
start()Schedule the timeout This is called on construction so it should not be called explicitly unless thetimer has been canceled
gThreadafter()Call after the thread has shut down
gThreadbefore()Called at the beginning of start()
gThreadextra_shutdown_steps = 0
gThreadextra_startup_steps = 0
gThreadg = NoneGreenlet instance of the thread set when the thread is started
24 Chapter 3 API Reference
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
gThreadjoin(timeout=None)Wait until the thread exits
Parameters timeout ndash Timeout in seconds (intfloat)
Raises eventletTimeout if the thread canrsquot be joined before the provided timeout
gThreadjoinable = TrueSet this to False if it is not possible to join the thread
gThreadkill()Kill the green thread
gThreadlogger_name
gThreadname = NoneName of the thread used in logs and such
gThreadping(timeout=None)
gThreadrespond_to_ping()
gThreadrun()
gThreadshould_stop = FalseSet when the thread is requested to stop
gThreadspawn(fun args kwargs)
gThreadstart()Spawn green thread and returns GreenThread instance
gThreadstart_periodic_timer(interval fun args kwargs)Apply function every interval seconds
Parameters
bull interval ndash Interval in seconds (intfloat)
bull fun ndash The function to apply
bull args ndash Additional arguments to pass
bull kwargs ndash Additional keyword arguments to pass
Returns entry object with cancel and kill methods
gThreadstop(join=True timeout=1e+100)Shutdown the thread
This will also cancel+kill any periodic timers registered by the thread
Parameters
bull join ndash Given that the thread is joinable if true will also wait until the thread exits (bycalling join())
bull timeout ndash Timeout for join (default is 1e+100)
312 cymebranchintsup
cymebranchintsup
bull Internal supervisor used to ensure our threads are still running
312 cymebranchintsup 25
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
class cymebranchintsupgSup(thread signal interval=5 timeout=600)
logger_name
run()
start_wait_child()
stop()
313 cymeapiviews
314 cymeapiweb
cymeapiweb
bull Contains utilities for creating our HTTP API
class cymeapiwebApiView(kwargs)
Accepted(args kwargs)
Created(data args kwargs)
NotImplemented(args kwargs)
Ok(data args kwargs)
Response(args kwargs)
dispatch(request args kwargs)
get_or_post(key default=None)
get_param(key type=None)
nowait = False
params(keys)
typemap = lttype lsquointrsquogt ltfunction ltlambdagt at 0x4a84ed8gt lttype lsquofloatrsquogt ltfunction ltlambdagt at 0x4a84f50gt
class cymeapiwebHttpResponseNotImplemented(content=rsquolsquo mimetype=None status=Nonecontent_type=None)
The requested action is not implemented Used for async requests when the operation is inherently sync
status_code = 501
class cymeapiwebHttpResponseTimeout(content=rsquolsquo mimetype=None status=None con-tent_type=None)
The operation timed out
status_code = 408
cymeapiwebJsonResponse(data status=200 access_control=None kwargs)Returns a JSON encoded response
cymeapiwebset_access_control_options(response options=None)
cymeapiwebsimple_get(fun)
26 Chapter 3 API Reference
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
315 cymemodels
cymemodels
class cymemodelsBroker(args kwargs)Broker connection arguments
urlAMQP (kombu) url
connection()Return a new Connection to this broker
as_dict()Returns this broker as a dictionary that can be Json encoded
poolA connection pool with connections to this broker
objectsThe manager for this model is BrokerManager
class cymemodelsQueue(args kwargs)An AMQP queue that can be consumed from by one or more instances
nameQueue name (unique max length 128)
exchangeExchange name (max length 128)
exchange_typeExchange type (max length 128)
routing_keyRouting keybinding key (max length 128)
optionsAdditional JSON encoded queueexchangebinding options see kombucompat for a list of optionssupported
is_enabledNot in use
created_atTimestamp created
as_dict()Returns dictionary representation of this queue that can be Json encoded
objectsThe manager for this model is QueueManager
class cymemodelsInstance(args kwargs)A celeryd instance
nameName of the instance
queuesQueues this instance should consume from (many to many relation to Queue)
315 cymemodels 27
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
max_concurrencyAutoscale setting for max concurrency (maximum number of processesthreadsgreen threads when theworker is active)
min_concurrencyAutoscale setting for min concurrency (minimum number of processesthreadsgreen threads when theworker is idle)
is_enabledFlag set if this instance should be running
created_atTimestamp of when this instance was first created
brokerThe broker this instance should connect to (foreign key to Broker)
BrokerBroker model class used (default is Broker)
QueueQueue model class used (default is Queue)
MultiToolClass used to startstop and restart celeryd instances (Default iscelerybinceleryd_multiMultiTool)
objectsThe manager used for this model is InstanceManager
cwdWorking directory used by all instances (Default is varruncyme)
as_dict()Returns dictionary representation of this instance that can be Json encoded
enable()Enables this instance (model-only)
disable()Disables this instance (model-only)
start(kwargs)Starts the instance
stop(kwargs)Shuts down the instance
restart(kwargs)Restarts the instance
alive(kwargs)Returns True if the pid responds to signals and the instance responds to ping broadcasts
stats(kwargs)Returns instance statistics (like celeryctl inspect stats)
autoscale(max=None min=None kwargs)Set maxmin autoscale settings
responds_to_ping(kwargs)Returns True if the instance responds to broadcast ping
28 Chapter 3 API Reference
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
responds_to_signal()Returns True if the pid file exists and the pid responds to signals
consuming_from(kwargs)Returns the queues the instance is currently consuming from
add_queue(q kwargs)Add queue for this instance by name
cancel_queue(queue kwargs)Cancel queue for this instance by Queue
getpid()Get the process id for this instance by reading its pid file
Returns None if the pid file does not exist
_action(action multi=rsquoceleryd-multirsquo)Execute celeryd-multi command
_query(cmd args= kwargs)Send remote control command and wait for this instances reply
316 cymemodelsmanagers
cymemanagers
bull These are the managers for our models in cymemodels
bull They are not to be used directly but accessed through the objects attribute of a Model
class cymemodelsmanagersAppManager
Brokers
add(name=None broker=None arguments=None extra_config=None)
from_json(name=None broker=None)
get_broker(url)
get_default()
instance(name=None broker=None)
recreate(name=None broker=None arguments=None extra_config=None)
class cymemodelsmanagersBrokerManager
default_url
get_default()
class cymemodelsmanagersInstanceManager
add(name=None queues=None max_concurrency=1 min_concurrency=1 broker=None pool=Noneapp=None arguments=None extra_config=None)
add_queue_to_instances(queue query)
disable(name)
316 cymemodelsmanagers 29
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
enable(name)
enabled()
remove(name)
remove_queue_from_instances(queue query)
class cymemodelsmanagersQueueManager
add(name exchange=None exchange_type=None routing_key=None options)
enabled()
317 cymestatus
cymestatus
class cymestatusStatus
all_instances()
ib(fun args kwargs)Shortcut to selfinsured( funim_self fun(args kwargs))
insured(instance fun args kwargs)Ensures any function performing a broadcast command completes despite intermittent connection failures
pause()
paused = False
respond_to_ping()
restart_all()
restart_max_rate = lsquo100srsquo
resume()
shutdown_all()
start_all()
318 cymetasks
cymetasks
bull We have our own version of the webhook task
bull It simply forwards the original request not depending on any semantics present in the query string of the requestnor in the data returned in the response
cymetasksDEFAULT_HEADERS = lsquoUser-Agentrsquo lsquoCelerycyme v004rsquoDefault HTTP headers to pass to the dispatched request
cymetasksUA = lsquoCelerycyme v004rsquoCyme User Agent string
cymetasksresponse_to_dict(r)
30 Chapter 3 API Reference
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
319 cymemanagementcommandscyme
bull cymendash Options
3191 cyme
Cyme management utility
Options
-a -appApplication to use Required for all operations except for when creating deleting or listing apps
-n -nowaitDonrsquot want for operations to complete (async)
-F -formatOutput format pretty (default) or json
-L -localFlag that if set means that operations will be performed locally for the current branch only This means theinstance directory must be properly set
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom logfile path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instances) Must be readable by the current user
This needs to be properly specified if the -L option is used
-b -brokerBroker to use for a local branches request
class cymemanagementcommandscymeCommand(env=None args kwargs)
args = lsquotype command [args]nEgn cyme appsn cyme appsadd ltnamegt [broker URL] [arguments] [extra config]n cyme apps[get|delete] ltnamegtnn cyme -a ltappgt instancesn cyme -a ltappgt instancesadd [name] [broker URL] [arguments] [extra config]n cyme -a ltappgt instances[get|delete|stats] ltnamegtn cyme -a ltappgt instancesautoscale ltnamegt [max] [min]nn cyme -a ltappgt queuesn cyme -a ltappgt queuesadd ltnamegt [exchange] [type] [rkey] [opts]n cyme -a ltappgt queues[get|delete] ltnamegtnn cyme start-alln cyme shutdown-allnn cyme createsuperusernn cyme shelln lsquo
create_superuser()
drop_into_shell()
handle(args kwargs)
help = lsquoCyme management utilityrsquo
name = lsquocymersquo
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x52cb098 -andashappgt ltOption at 0x52cb3b0 -nndashnowaitgt ltOption at 0x55c64d0 -Fndashformatgt ltOption at 0x52cfe60 -Lndashlocalgt ltOption at 0x560f5f0 -lndashloglevelgt ltOption at 0x56037a0 -Dndashinstance-dirgt ltOption at 0x5603998 -undashurlgt ltOption at 0x5603368 -bndashbrokergt)
restart_all()
shutdown_all()
319 cymemanagementcommandscyme 31
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
start_all()
status
class cymemanagementcommandscymeI(app=None format=None nowait=False url=Nonekwargs)
DISPATCH(fqdn args)
add_app(args kwargs)
add_consumer(args kwargs)
add_instance(args kwargs)
add_queue(args kwargs)
all_apps(args kwargs)
all_consumers(args kwargs)
all_instances(args kwargs)
all_queues(args kwargs)
delete_app(args kwargs)
delete_consumer(args kwargs)
delete_instance(args kwargs)
delete_queue(args kwargs)
format_optargs(optargs)
format_response(ret)
get_app(args kwargs)
get_instances(args kwargs)
get_queue(args kwargs)
getsig(fun opt_args=None)
instance_autoscale(args kwargs)
instance_stats(args kwargs)
prepare_response(ret)
class cymemanagementcommandscymeLocalI(args kwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
32 Chapter 3 API Reference
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app()
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
class cymemanagementcommandscymeWebI(app=None format=None nowait=False url=Nonekwargs)
add_app(name broker=None arguments=None extra_config=None)
add_consumer(instance_name queue_name)
add_instance(name=None broker=None arguments=None extra_config=None)
add_queue(name exchange=None exchange_type=None routing_key=None options=None)
all_apps()
all_branches()
all_consumers(instance_name)
all_instances()
all_queues()
client
delete_app(name)
delete_consumer(instance_name queue_name)
delete_instance(name)
delete_queue(name)
get_app(name)
get_instance(name)
get_queue(name)
instance_autoscale(name max=None min=None)
instance_stats(name)
prepare_response(ret)
320 cymemanagementcommandscyme_branch
bull cyme-branchndash Options
320 cymemanagementcommandscyme_branch 33
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
3201 cyme-branch
Starts the cyme branch service
Options
-i -idSet branch id if not provided one will be automatically generated
-without-httpdDisable the HTTP server thread
-l -loglevelSet custom log level One of DEBUGINFOWARNINGERRORCRITICAL Default is INFO
-f -logfileSet custom log file path Default is ltstderrgt
-D -instance-dirCustom instance directory (default is instanceslsquo) Must be writeable by the user cyme-branch runs as
-C -numcNumber of controllers to start to handle simultaneous requests Each controller requires one AMQP connectionDefault is 2
-sup-intervalSupervisor schedule Interval in seconds Default is 5
class cymemanagementcommandscyme_branchCommand(env=None args kwargs)
args = lsquo[optional port number or ipaddrport]rsquo
banner()
branch_cls = lsquocymebranchBranchrsquo
connect_signals()
default_detach_logfile = lsquob ranchlogrsquo
default_detach_pidfile = lsquobranchpidrsquo
handle(args kwargs)
help = lsquoStarts a cyme branchrsquo
install_signal_handlers()
name = lsquocyme-branchrsquo
on_branch_ready(sender=None kwargs)
on_branch_shutdown(sender=None kwargs)
option_list = (ltOption at 0x2d62320 -vndashverbositygt ltOption at 0x2d623b0 ndashsettingsgt ltOption at 0x2d62368 ndashpythonpathgt ltOption at 0x2d623f8 ndashtracebackgt ltOption at 0x5937950 -bndashbrokergt ltOption at 0x5a4ecf8 ndashdetachgt ltOption at 0x59341b8 -indashidgt ltOption at 0x59318c0 -Xndashno-interactiongt ltOption at 0x58fdd40 ndashwithout-httpdgt ltOption at 0x58fdf80 -lndashloglevelgt ltOption at 0x58fddd0 -Dndashinstance-dirgt ltOption at 0x5a587e8 -Cndashnumcgt ltOption at 0x5a58758 ndashsup-intervalgt ltOption at 0x5a581b8 -fndashlogfilegt ltOption at 0x5a582d8 ndashpidfilegt ltOption at 0x5a58098 ndashuidgt ltOption at 0x5a58878 ndashgidgt ltOption at 0x5a58830 ndashumaskgt ltOption at 0x5a58170 ndashworkdirgt)
repr_controller_id(c)
set_process_title(info)
setup_default_env(env)
setup_shutdown_progress(sender=None kwargs)
setup_startup_progress(sender=None kwargs)
34 Chapter 3 API Reference
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
signals
stop()
321 cymebinbase
cymebinbase
bull Utilities used by command line applications basically just to set up Django without having an actual project
class cymebinbaseBaseApp
env = None
get_version()
instance_dir = None
needs_eventlet = False
run_from_argv(argv=None)
class cymebinbaseEnv(needs_eventlet=False instance_dir=None)
configure()
management
setup_eventlet()
setup_pool_limit(kwargs)
syncdb(interactive=True)
cymebinbaseapp(attrs)
322 cymebincyme
cymebincyme
323 cymebincyme_branch
cymebincyme_branch
bull This is the script run by the cyme-branch script installed by the Cyme distribution (defined in setuppyrsquosentry_points)
bull It in turn executes the cyme-branch management command
324 cymeutils
cymeutils
321 cymebinbase 35
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
class cymeutilsLazyProgressBar(size description=None endtext=None)
finish(kwargs)
step(i=1 kwargs)
class cymeutilsPathunipathPath version that can use the operator to combine paths
gtgtgt p = Path(foo)gtgtgt p bar bazPath(foobarbaz)
cymeutilsfind_package(mod _s=None)Find the package a module belongs to
if you have structure
package__init__pyfoopybar__init__pybarbazpy
Then the following examples returns
gtgtgt find_package(import_module(package))packagegtgtgt find_package(import_module(packagefoo))packagegtgtgt find_package(import_module(packagebarbaz))gtgtgt packagebar
Note that this does not look at the file system but rather uses the __package__ attribute of a module
cymeutilsfind_symbol(origin sym)Find symbol in module relative by origin
Eg if origin is an object in the module packagefoo then
gtgtgt find_symbol(origin barmy_symbol)
will return the object my_symbol from module packagebar
cymeutilsforce_list(obj)Force object to be a list
If obj is a scalar value then a list with that value as sole element is returned or if obj is a tuple then it iscoerced into a list
cymeutilsimerge_settings(a b)Merge two django settings modules keys in b have precedence
cymeutilsinstantiate(origin sym args kwargs)Like find_symbol() but instantiates the class found using args and kwargs
cymeutilsredirect_stdouts_to_logger(loglevel=rsquoINFOrsquo logfile=None redi-rect_level=rsquoWARNINGrsquo stdout=False stderr=True)
See celerylogLogredirect_stdouts_to_logger()
cymeutilssetup_logging(loglevel=rsquoINFOrsquo logfile=None)Setup logging using loglevel and logfile
stderr will be used if not logfile provided
36 Chapter 3 API Reference
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
CHAPTER
FOUR
INDICES AND TABLES
bull genindex
bull modindex
bull search
37
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
Cyme Documentation Release 004
38 Chapter 4 Indices and tables
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39
PYTHON MODULE INDEX
ccymeapiweb cymebinbase cymebincyme cymebincyme_branch cymebranch cymebranchhttpd cymebranchintsup cymebranchmanagers cymebranchmetrics cymebranchsignals cymebranchstate cymebranchsupervisor cymebranchthread cymeclient cymeclientbase cymemanagementcommandscyme cymemanagementcommandscyme_branch
cymemodels cymemodelsmanagers cymestatus cymetasks cymeutils
39