HEALTHCHECK of a Docker container running Celery tasks?

Question

I know one of the ways to check health for Docker container is using the commmand

HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1

But in case of workers there is no such URL to hit , How to check the container's health in that case ?

Solution 1

The celery inspect ping command comes in handy, as it does a whole trip: it sends a "ping" task on the broker, workers respond and celery fetches the responses.

Assuming your app is named tasks.add, you may ping all your workers:

/app $ celery inspect ping -A tasks.add
-> [email protected]: OK
        pong
-> [email protected]: OK
        pong

With aa7c21dd0e96 being the Docker hostname, and thus available in $HOSTNAME.

To ping a single node, you would have to run:

celery inspect ping -A tasks.add -d [email protected]$HOSTNAME

Here, d stands for destination.

The line to add to your Dockerfile:

HEALTHCHECK CMD celery inspect ping -A tasks.add -d [email protected]$HOSTNAME

---

Sample outputs:

/app $ celery inspect ping -A tasks.add -d fake_node
Error: No nodes replied within time constraint.
/app $ echo $?
69

Unhealthy if the node does not exist or does not reply

/app $ celery inspect ping -A tasks.add -d [email protected]$HOSTNAME
-> [email protected]: OK
        pong
/app $ echo $?
0

Healthy when the node replies pong.

/app $ celery inspect ping -d [email protected]$HOSTNAME
Traceback (most recent call last):
...
    raise socket.error(last_err)
OSError: [Errno 111] Connection refused
/app $ echo $?
1

Unhealthy when the broker is not available - I removed the app, so it tries to connect to a local AMPQ and fails This might not suit your needs, the broker is unhealthy, not the worker.

Solution 2

The below example snippet, derived from that posted by @PunKeel, is applicable for those looking to implement health check in docker-compose.yml which could be used through docker-compose or docker stack deploy.

worker:
    build:
        context: .
        dockerfile: Dockerfile
    image: myimage
    links:
        - rabbitmq
    restart: always
    command: celery worker --hostname=%h --broker=amqp://rabbitmq:5672
    healthcheck:
        test: celery inspect ping -b amqp://rabbitmq:5672 -d [email protected]$$HOSTNAME
        interval: 30s
        timeout: 10s
        retries: 3

Notice the extra $ in the command, so that $HOSTNAME actually gets passed into the container. I also didn't use the -A flag.

Ideally, rabbitmq should also have its own health check, perhaps with curl guest:[email protected]:15672/api/overview, since docker wouldn't be able to discern if worker is down or the broker is down with celery inspect ping.

Comments

[EsseTi]

with SQS, how do you do? you can't use the result with sqs and celery docs.celeryproject.org/en/master/getting-started/brokers/…

[silviot]

I believe "Assuming your app is named tasks.add" it's a confusing assumption: an app should not live in a module named tasks, if you want to minimize confusion.

[SomeGuyOnAComputer]

For SQS, you can specify the pidfile using --pidfile /opt/celery/celery_pid and check that the process is still running using kill -0 $(cat /opt/celery/celery_pid). Got this from the guy that runs masterpoint.io.

[luckydonald]

In case you are using authentication, you can provide username and password by writing user:[email protected]:5672. You can even use environment variables you set yourself for your application like, celery inspect ping -b amqp://$${RABBITMQ_USER}:$${RABBITMQ_PASS}@$${RABBITMQ_HOST}‌​:5672 -d [email protected]$$HOSTNAME

[Martin Thoma]

On Celery 5.2.3: Error: No such option: -b

[Martin Thoma]

The order needs to be changed: celery -b amqp://rabbitmq:5672 inspect ping

[pangyuteng]

@SomeGuyOnAComputer nice tip on using kill -0 thanks!

[pangyuteng]

updated healthheck test command per @Mart!