Celery workers unable to connect to redis on docker instances

Question

I have a dockerized setup running a Django app within which I use Celery tasks. Celery uses Redis as the broker.

Versioning:

• Docker version 17.09.0-ce, build afdb6d4
• docker-compose version 1.15.0, build e12f3b9
• Django==1.9.6
• django-celery-beat==1.0.1
• celery==4.1.0
• celery[redis]
• redis==2.10.5

Problem:

My celery workers appear to be unable to connect to the redis container located at localhost:6379. I am able to telnet into the redis server on the specified port. I am able to verify redis-server is running on the container.

When I manually connect to the Celery docker instance and attempt to create a worker using the command celery -A backend worker -l info I get the notice:

[2017-11-13 18:07:50,937: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 99 connecting to localhost:6379. Cannot assign requested address.. Trying again in 4.00 seconds...

Notes:

I am able to telnet in to the redis container on port 6379. On the redis container, redis-server is running.

Is there anything else that I'm missing? I've gone pretty far down the rabbit hole, but feel like I'm missing something really simple.

DOCKER CONFIG FILES:

docker-compose.common.yml here
docker-compose.dev.yml here

Solution 1

When you use docker-compose, you aren't going to be using localhost for inter-container communication, you would be using the compose-assigned hostname of the container. In this case, the hostname of your redis container is redis. The top level elements under services: are your default host names.

So for celery to connect to redis, you should try redis://redis:6379/0. Since the protocol and the service name are the same, I'll elaborate a little more: if you named your redis service "butter-pecan-redis" in your docker-compose, you would instead use redis://butter-pecan-redis:6379/0.

Also, docker-compose.dev.yml doesn't appear to have celery and redis on a common network, which might cause them not to be able to see each other. I believe they need to share at least one network in common to be able to resolve their respective host names.

Networking in docker-compose has an example in the first handful of paragraphs, with a docker-compose.yml to look at.

Solution 2

You may need to add the link and depends_on sections to your docker compose file, and then reference the containers by their hostname.

Updated docker-compose.yml:

version: '2.1'
services:
    db:
        image: postgres
    memcached:
        image: memcached
    redis:
        image: redis
        ports:
          - '6379:6379'
    backend-base:
        build:
            context: .
            dockerfile: backend/Dockerfile-base
        image: "/backend:base"
    backend:
        build:
            context: .
            dockerfile: backend/Dockerfile
        image: "/backend:${ENV:-local}"
        command: ./wait-for-it.sh db:5432 -- gunicorn backend.wsgi:application -b 0.0.0.0:8000 -k gevent -w 3
        ports:
            - 8000
        links:
            - db
            - redis
            - memcached
        depends_on:
            - db
            - redis
            - memcached
    celery:
        image: "/backend:${ENV:-local}"
        command: ./wait-for-it.sh db:5432 -- celery worker -E -B --loglevel=INFO --concurrency=1
        environment:
            C_FORCE_ROOT: "yes"
        links:
            - db
            - redis
            - memcached
        depends_on:
            - db
            - redis
            - memcached
    frontend-base:
        build:
            context: .
            dockerfile: frontend/Dockerfile-base
            args:
                NPM_REGISTRY: http://.view.build
                PACKAGE_INSTALLER: yarn
        image: "/frontend:base"
        links:
            - db
            - redis
            - memcached
        depends_on:
            - db
            - redis
            - memcached
    frontend:
        build:
            context: .
            dockerfile: frontend/Dockerfile
        image: "/frontend:${ENV:-local}"
        command: 'bash -c ''gulp'''
        working_dir: /app/user
        environment:
            PORT: 3000
        links:
            - db
            - redis
            - memcached
        depends_on:
            - db
            - redis
            - memcached

Then configure the urls to redis, postgres, memcached, etc. with:

• redis://redis:6379/0
• postgres://user:pass@db:5432/database

Comments

[Admin]

They did not share a common network, which helped resolve the issue.

[Javier Enciso]

Hey, thanks for your contribution. What do you mean exactly by "...configure the urls to redis, postgres, memcached..."?

[AATHITH RAJENDRAN]

if two of my containers are in the same network, this redis://butter-pecan-redis:6379/0 way of connecting still work??(I don't want to use docker-compose)

[bluescores]

@AATHITHRAJENDRAN No, it won't. Docker inherits the DNS of the host, so unless you have a DNS entry for butter-pecan-redis mapped to localhost:<port> somewhere outside of Docker, one container won't resolve another container by name. Docker-compose does something extra that allows this to work, I can't remember if it's something with a DNS per Docker network in compose, or if docker-compose adjusts hosts or resolv.conf. Either way, the simple by-name resolution is a docker-compose feature.

[Jarad]

This was the missing piece of my puzzle to finally getting Docker, Django, Celery and Redis to work together from a docker-compose.yml file. Thanks for explaining this!