An enterprise scheduler for python (like quartz)

Question

I am looking for an enterprise tasks scheduler for python, like quartz is for Java. Requirements:

• Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and must be fired after restarting.
• Jobs must enter and exit the scheduler in a transaction (i.e. if some database operation fail, in a database unrelated to the scheduler, then the job must not have exited or entered the scheduler).
• Scalability. Depends on the measure of success of the project, but I would prefer to know from the beggining that I am not starting from a dead end.
• Configurability: when tasks expire, how many can be fired simultaneously, etc.

Any recommendations? Is there something python specific, or is it possible (or even good) to interface to Quartz from python.

Solution 1

Is APScheduler what you're looking for?

Solution 2

You can use Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Install Celery using pip install celery

Another option is using RQ.

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.

Install using pip install rq.

Solution 3

We're using Sun Grid Engine, which is accessible through DRMAA, which happens to have Python bindings.