Create celery tasks then run synchronously

Question

My app gathers a bunch of phone numbers on a page. Once the user hits the submit button I create a celery task to call each number and give a reminder message then redirect them to a page where they can see the live updates about the call. I am using web sockets to live update the status of each call and need the tasks to execute synchronously as I only have access to dial out from one number.

So once the first call/task is completed, I want the next one to fire off.

I took a look at CELERY_ALWAYS_EAGER settings but it just went through the first iteration and stopped.

@task
def reminder(number):
    # CODE THAT CALLS NUMBER HERE....

def make_calls(request):
    for number in phone_numbers:                     
        reminder.delay(number)      

    return redirect('live_call_updates') 

Solution 1

if you want to fire each call one after another, why dont you wrap all the calls in one task

@task
def make_a_lot_of_calls(numbers):
    for num in numbers:
        # Assuming that reminder blocks till the call finishes
        reminder(number)

def make_calls(request):
    make_a_lot_of_calls.delay(phone_numers)                          
    return redirect('live_call_updates') 

Solution 2

If you look at the celery DOCS on tasks you see that to call a task synchronosuly, you use the apply() method as opposed to the apply_async() method.

So in your case you could use:

 reminder.apply(args=[number])

The DOCS also note that:
If the CELERY_ALWAYS_EAGER setting is set, it will be replaced by a local apply() call instead.

Thanks to @JivanAmara who in the comments reiterated that when using apply(), the task will run locally(in the server/computer in which its called). And this can have ramifications, if you intended to run your tasks across multiple servers/machines.

Solution 3

Can use celery chain.

from celery import chain
tasks = [reminder.s(number) for number in phone_numbers]
chain(*tasks).apply_async()

Comments

[srj]

do you have a celery task per phone number, or are you using celery to asynchronously start the calls one by one?

[JivanAmara]

Keep in mind that running a task using apply() will run synchronously, but also will run locally. This is a problem if you intend to have multiple machines doing the processing.

[JivanAmara]

To rewrite the task for use with celery may be appropriate at times, but here it seems making a single call is an appropriate definition for a task.