passing django request object to celery task

Question

I have a task in tasks.py like so:

@app.task
def location(request):
....

I am trying to pass the request object directly from a few to task like so:

def tag_location(request):
    tasks.location.delay(request)
    return JsonResponse({'response': 1})

I am getting an error that it can't be serialized i guess? How do I fix this? trouble is I have file upload objects as well .. its not all simple data types.

Solution

Because the request object contains references to things which aren't practical to serialize — like uploaded files, or the socket associated with the request — there's no general purpose way to serialize it.

Instead, you should just pull out and pass the portions of it that you need. For example, something like:

import tempfile

@app.task
def location(user_id, uploaded_file_path):
    # … do stuff …

def tag_location(request):
    with tempfile.NamedTemporaryFile(delete=False) as f:
        for chunk in request.FILES["some_file"].chunks():
            f.write(chunk)
    tasks.location.delay(request.user.id, f.name)
    return JsonResponse({'response': 1})

Comments

[Tyler]

ok i'll try this... thats what I was looking for wasn't sure how to pass the files

[Tyler]

actually does this work? I thought on heroku you can't save files like that... also how do I retrieve the file data in the task with the file path

[Tyler]

it seems like this won't always work because that temp file isn't guarenteed to stay across other processes stackoverflow.com/questions/12416738/…

[David Wolever]

Ah yes, this won't work on Heroku. On Heroku you'll need to save the uploaded file to S3 or some other data store, then pass the URL to the worker.

[Tyler]

I ended up reading the file with request.FILES["some_file"].read() and sending that to the task. Thanks for the help this got me going in the right direction!

[David Wolever]

No problem! Just be mindful of that approach, though, as it may seriously bog things down if there is a large file upload (your task queue may not be too happy with multi-megabyte jobs).

[Tyler]

we have everything compresses down to less than a megabyte at all times so hopefully that will work... I'll be doing some load testing eventually so I guess it'll come up there. Thanks!