What is the best option for a (Python 3) task queue on Windows now that Celery 4 has dropped Windows support?

Question

We run a Flask site under IIS on Windows, and for out-of-process tasks we use Celery. Celery has given us some problems under Windows, but for now we are satisfied running version 3.1.12, using RabbitMQ/AMQP as a back-end, which works under Windows.

The new version of Celery (4) has dropped support for Windows, so I'm looking for a viable alternative.

RQ seems a very nice task queue, but it also does not support Windows (bottom of the page)

I have seen some more, seemingly less popular task queues like:

• kuyruk
• tasktiger
• huey

But it's unclear if these support Windows and Flask. I'm wondering if anyone has experience running a Python task queue under Windows which works. Maybe one of the ones I mentioned, or an alternative.

It's not an option for us to run a Linux machine, because we have no experience administering Linux, and we have a lot of legacy stuff running that requires Windows.

Solution

I run Flask with Huey on Windows without any issues, admittedly only for development and testing. For production I use Flask/Huey on Linux servers. Both with the Redis back-end, Flask 0.12 and Huey 1.2.0 .

I use the factory pattern to create a specialised "cut down" version of a Flask app for specific use by Huey tasks. This version doesn't load blueprints or configure Flask-Admin as these aren't required in the Huey tasks.

Example code of __init__.py in app folder. App is a class extending from Flask:

def create_app(settings_override=None):

    app = App('app')

    if settings_override:
        app.config.from_object(settings_override)
    else:
        app.config.from_object(os.environ['APP_SETTINGS'])

    from .ext import configure_extensions
    configure_extensions(app, admin, load_modules=True)

    # REST
    import rest.api_v1
    app.register_blueprint(api_v1_bp, url_prefix='/api/v1')

    #  ... and more suff


def create_huey_app():
    app = App('huey app')

    app.config.from_object(os.environ['APP_SETTINGS'])

    from .ext import configure_extensions
    configure_extensions(app, admin=None, load_modules=False)

    return app

The idea of configure_extensions is taken from Quokka CMS. Examine its app __init__.py and its extensions module to see how this is implemented. Note also how this project too creates a specific app (create_celery_app) for use with the Celery task queue.

Example of tasks.py. Note the use of with app.app_context(): to create the Flask context. Now my functions have access to the extensions such as Flask-Mail, Flask-SqlAlchemy (db, models) etc.

@huey.task()
def generate_transaction_documents_and_email(transaction_id):
    app = create_huey_app()
    with app.app_context():
        reports.generate_transaction_documents_and_email(transaction_id)


@huey.task()
def send_email(subject, recipients, text_body, html_body, attachments=[], cc=[]):
    app = create_huey_app()
    with app.app_context():
        emails.send_email(subject, recipients, text_body, html_body, attachments, cc)


@huey.periodic_task(crontab(minute='30'))
def synchronize_mailing_list():
    app = create_huey_app()
    if app.config['CREATESEND_SYNCHRONIZE']:
        _list_name = app.config['CREATESEND_LIST']
        with app.app_context():
            sync_delete_ar_subscribers(_list_name)
            sync_add_ar_subscribers(_list_name)

Comments

[Kevin Vasko]

Did you ever find one? I've been looking for one for the past 2 weeks and can't find one that is still developed.

[Erik Oosterwaal]

@KevinVasko; Huey is the one used most widely it seems. I have been experimenting with it, and it is simple and it works. However, I have found one big issue, and the author couldn't help me with it either; I'm unable to use Huey workers that need a Flask context. I can't figure out how to pass the tasks this context. This stops me currently using Huey.

[Kevin Vasko]

Thanks for feedback, I'll check out Huey, I'm not having to work with Flask so hopefully that won't be a problem. I also found one called "WorQ", but it hasn't been updated in 3 years or so and it isn't clear if it support Windows or not.

[Erik Oosterwaal]

Huey uses Redis as a backend, of which there is a Windows port. It runs fine on itself. Pity about the Flask integration. I'm trying to figure that out, but I might have to abandon Huey alltogether. Good luck!

[SteveJ]

another option is django-channels. Channels has a 'delay server', so you can implement scheduled tasks as well.

[C. Yduqoli]

Huey seems to be the only one which kind of supports Windows. But only the threaded, not the multiple process backend. Which is useless for CPU bound tasks because of the GIL.

[Erik Oosterwaal]

@C.Yduqoli is that true? I was having troubles with different backends, but I thought it was my code. Do you have a source for that? Is it somewhere in the documentation?

[C. Yduqoli]

@ErikOosterwaal github.com/coleifer/huey/issues/323 Here the developer says "Multiprocessing doesn't work on Windoze." Although it is not stated anywhere clearly, I think Windows support for huey was never intended.

[Erik Oosterwaal]

Thank you for this info; I will try to set this up for myself and come back to this question afterwards.

[Erik Oosterwaal]

I'm implementing this, but I'm unsure what you do inside the configure_extensions import. Would you consider sharing what goes on in there?

[Erik Oosterwaal]

Thanks! I implemented it, still a little rough around the edges, but it works. I took the rest of the setup from the huey docs; created a config.py and let both the views I use to launch a task as well as the decorators for the tasks themselves import from that. I now have Flask interface to launch out of process tasks and a worker that runs from the same codebase.