Is it possible to integrate celery with Kafka

Question

I want to introduce multiprocessing in my code using celery. But currently my queue implementation is in Kafka.

Currently celery website mentions of only these 4 brokers: http://docs.celeryproject.org/en/master/getting-started/brokers/index.html#broker-overview

Is it possible to integrate Celery with Kafka something similar to RabbitMQ mentioned below:

from celery import Celery
app = Celery('tasks', broker='pyamqp://guest@localhost//')
@app.task
def add(x, y):
    return x + y

Solution 1

There was discussion on having kafka as broker for celery, And there is a partial implementation of it. I hope this helps you

https://github.com/celery/kombu/issues/301

Looks like it will be available this Oct 31 or so.

https://github.com/celery/kombu/milestone/2

Solution 2

As you are searching for multiprocessing solution for Kafka I would recommend to take a look at Faust library.

Here is an example (very close to yours) in comparison with Celery:

https://faust.readthedocs.io/en/latest/playbooks/vscelery.html

import faust

app = faust.App('myapp', broker='kafka://')

@app.agent()
async def add(stream):
    async for op in stream:
        yield op.x + op.y

Also one of the creators of Faust is the author of Celery.

I am telling that because at this moment already 7 years have passed since the creation of the ticket mentioned by @user3382968, and still without progress.

Comments

[Kristoffer Bakkejord]

It seems like support for Kafka has been postponed to a later version since this answer. I suggest checking which milestone issue #301 belongs to.

[TheTechRobo Stands for Ukraine]

Please add an example of how to use it. Links can change, and so it's best to add a small snippet yourself.

[vilozio]

@TheTechRobo36414519 thank you, I added an example of usage.

[Touisteur EmporteUneVache]

Hi, I know, old question, but it seems kafka support has been merged 28 days ago github.com/celery/kombu/pull/1506

[AntonOfTheWoods]

Faust has been abandoned. It was forked to faust-streaming but that has also been abandoned. It is not compatible with the future kraft version of kafka and will never be updated to support it.