Celery Heartbeat Not Working

Question

I have set heartbeat in Celery settings:

BROKER_HEARTBEAT = 10

I have also set this configuration value in RabbitMQ config:

'heartbeat' => '10',

But somehow heartbeats are still disabled:

ubuntu@sync1:~$ sudo rabbitmqctl list_connections name timeout
Listing connections ...
some_address:37781 -> other_address:5672    0
some_address:37782 -> other_address:5672    0
...done.

Any ideas what am I doing wrong?

UPDATE:

So now I get:

ubuntu@sync1:/etc/puppet$ sudo rabbitmqctl list_connections name timeout
Listing connections ...
some_address:41281 -> other_address:5672    10
some_address:41282 -> other_address:5672    10
some_address:41562 -> other_address:5672    0
some_address:41563 -> other_address:5672    0
some_address:41564 -> other_address:5672    0
some_address:41565 -> other_address:5672    0
some_address:41566 -> other_address:5672    0
some_address:41567 -> other_address:5672    0
some_address:41568 -> other_address:5672    0
...done.

I have 3 servers:

• RabbitMQ broker
• RESTful API server
• Remote Worker server

It appears the remote demonised Celery workers send heartbeats correctly. The RESTful API server using Celery to remotely process tasks is not using heartbeat for some reason.

Solution 1

the heartbeat of celery worker is application level heartbeat, not AMQP protocol's heartbeat. Each worker periodically send heartbeat event message to "celeryev" event exchange in BROKER. The heartbeat event is forwarded back to worker such worker can know the health status of BROKER. If number of loss heartbeat exceeding a threshold, the worker can do some reconnect action to BROKER.

For the rest of detail, you may check this page The section: BROKER_FAILOVER_STRATEGY describes the actions you may do for dropping from a BROKER.

Solution 2

Celery worker support AMQP heartbeat definitely. The configuration item BROKER_HEARTBEAT is used to define the heartbeat interval of AMQP client(celery worker). We can find the description of BROKER_HEARTBEAT here Celery Doc!

The possible causes of heartbeat not work:

1. Use a wrong transport such as 'librabbitmq' As celery doc described, only 'pyamqp' transport support BROKER_HEARTBEAT. We need to check whether if librabbitmq package is installed or we can use 'pyamqp' transport in broker url: 'pyamqp://userid:password@hostname:port/virtual_host' rather than 'amqp://userid:password@hostname:port/virtual_host'

2. No event send to celery worker during three heartbeat interval after boot up Check code here to see how heartbeat works! drain_events will be called during worker boot up, see code here! If there's no event sent to celery worker, connection.heartbeat_check will not be called.

By the way, connection.heartbeat_check is defined here!

Hopes to help someone encounter the heartbeat issue.

Comments

[odedfos]

The celery worker is also supposed to perform "real" heartbeats. You can see this in the code: github.com/celery/celery/blob/3.1/celery/worker/loops.py#L36 I could not find any indication to the heartbeats being performed via the "celeryev" queue, though as a side effect having those messages would keep the connection "alive" if heartbeat are enabled.

[Houcheng]

Nice to know that the celery supports this feature. Please check the heartbeat code here: github.com/celery/celery/blob/… The heartbeat timer uses the event dispatcher to send the "application level" heartbeat events.

[Houcheng]

And the event exchange is "celeryev". This is defined in the event.py: github.com/celery/celery/blob/…