Consumer Connection error with django and celery+rabbitmq?

Question

I'm trying to set up celeryd with django and rabbit-mq. So far, I've done the following:

• Installed celery from pip
• Installed rabbitmq via the debs available from their repository
• Added a user and vhost to rabbitmq via rabbitmqctl, as well as permissions for that user
• Started the rabbitmq-server
• Installed django-celery via pip
• Set up django-celery, including its tables
• Configured the various things in settings.py (BROKER_HOST, BROKER_PORT, BROKER_USER, BROKER_PASSWORD, BROKER_VHOST, as well as importing djecelery, calling the setup function and adding it to the INSTALLED APPS). I've double checked and all of these values are correct (at least, user, password and vhost are correct).

So now, when I run python manage.py celeryd -l info I get connection errors (see below). Anyone have any idea why?

$ python manage.py celeryd -l info
/usr/local/lib/python2.7/dist-packages/djcelery/loaders.py:108: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-05-15 18:38:04,486: WARNING/MainProcess]  

 -------------- celery@ubuntu v2.5.3
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      amqp://celeryuser@localhost:5672/celeryhost
- ** ----------   . loader:      djcelery.loaders.DjangoLoader
- ** ----------   . logfile:     [stderr]@INFO
- ** ----------   . concurrency: 1
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
 --------------   . celery:      exchange:celery (direct) binding:celery


[Tasks]


[2012-05-15 18:38:04,562: INFO/PoolWorker-1] child process calling self.run()
[2012-05-15 18:38:04,565: WARNING/MainProcess] celery@ubuntu has started.
[2012-05-15 18:38:07,572: ERROR/MainProcess] Consumer: Connection Error: [Errno 104] Connection reset by peer. Trying again in 2 seconds...
^C[2012-05-15 18:38:08,434: WARNING/MainProcess] celeryd: Hitting Ctrl+C again will terminate all running tasks!
[2012-05-15 18:38:08,435: WARNING/MainProcess] celeryd: Warm shutdown (MainProcess)
[2012-05-15 18:38:09,372: INFO/PoolWorker-1] process shutting down
[2012-05-15 18:38:09,373: INFO/PoolWorker-1] process exiting with exitcode 0
[2012-05-15 18:38:09,376: INFO/MainProcess] process shutting down

Solution 1

Your problem is in the BROKER_URL.

With an additional VHOST, the right config would be:

BROKER_URL='amqp://celeryuser@localhost:5672//'
BROKER_VHOST='/celeryhost'

Solution 2

For me, the following URL ending worked: ...@localhost:5672/celeryvhost

Solution 3

I had similar issue with RabbitMQ and problem was that my user doesn't have permission to create messages in RabbitMQ.

Try run the following script on your RabbitMQ server with guest user and if it creates a job try it with your user:

from celery import Celery

app = Celery('tasks', broker='amqp://radek:**@localhost:5672//')

@app.task
def add(x, y):
    return x + y

If you got the same error just set up the permission for your user:

rabbitmqctl set_permissions -p / radek ".*" ".*" ".*"

Solution 4

What does your BROKER_URL look like in settings.py?

By default RabbitMQ has a guest user, so if you can connect with

BROKER_URL = "amqp://guest:guest@localhost:5672//"

then the problem is your setup for RabbitMQs user, password, or virtualhost.

Solution 5

Your rabbitmq server must not be up and/or configured properly. Verify that it is and try again - or, better yet, if you are just trying to test something out and you are queue agnostic, take out rabbitmq and start using redis. It is much easier to configure.

I just cut-and-pasted this code from one of my project, and it works just fine:

import djcelery
from datetime import timedelta

djcelery.setup_loader()


BROKER_BACKEND = "redis"
BROKER_HOST = "localhost"
BROKER_PORT = 6379
BROKER_VHOST = "0"

CELERYD_LOG_LEVEL  = 'DEBUG'
CELERY_RESULT_BACKEND = "redis"
CELERY_TASK_RESULT_EXPIRES = 150000
REDIS_HOST = "localhost"
REDIS_PORT = 6379
REDIS_DB = "0"
CELERYD_CONCURRENCY = 1
CELERYD_MAX_TASKS_PER_CHILD = 4

CELERY_IMPORTS = (
    "apps.app1.tasks",
    "apps.app2.tasks",
)

View more solutions