ImportError: cannot import name celery

Question

I'm trying to run some background jobs using Celery + Redis + Flask.

My app structure is:

myapp/
  celery_worker.py
  manage.py
  myapp/
    __init__.py
    app.py
    bot/
      __init__.py
      tasks.py
    accounts/
      views.py

I initialise celery into app.py as:

celery = Celery('tasks', include=['bot.tasks'])

def create_app(config=None, app_name=None, blueprints=None):
    # more Celery and other config here like celery.conf.update()

And start worker by celery_worker.py

from myapp import create_app, celery

app = create_app()
app.app_context().push()

tasks.py contains

from myapp import celery

@celery.task(name='my_task_bot')
def my_task_bot():
     # pass

Now, when I try to access tasks from tasks.py into accounts/views.py with following import:

from ..bot.tasks import my_task_bot

I getting ImportError: cannot import name celery error.

No matter, where I try to import tasks, I'm getting this error. This celery instance seems to be the one initialised in app.py, but is not getting imported.

PS: I'm following pattern of https://github.com/ezequielo/flask_celery_exp repo and it works fine. But not my app.

Edit:

Here is Traceback:

Traceback (most recent call last):
  File "manage.py", line 8, in <module>
    from myapp import create_app
  File "/home/mars/myapp/myapp/__init__.py", line 3, in <module>
    from app import create_app, celery
  File "/home/mars/myapp/myapp/app.py", line 18, in <module>
    from .accounts import (accounts, AccountsAdmin) 
  File "/home/mars/myapp/myapp/accounts/__init__.py", line 7, in <module>
    from .views import accounts
  File "/home/mars/myapp/myapp/accounts/views.py", line 6, in <module>
    from ..bot.tasks import my_task_bot
  File "/home/mars/myapp/myapp/bot/tasks.py", line 14, in <module>
    from myapp import celery
ImportError: cannot import name celery

Fatal error: local() encountered an error (return code 1) while executing 'python manage.py initdb'

I get above error on trying to start any of either manage.py or celery_worker.

Please suggest.

Solution 1

Your import logic is not correct and leads to circular dependency. Do not initialize celery in app.py together with reference to the module (accounts) that in turn imports celery instance.

You have several options:

1. Remove dependency from app accounts at app.py module

2. Put celery initialization object into shared module that does not refer to any other modules, but referred by /*/tasks.py submodules.

3. Do not import celery in tasks, but use shared_task decorator. For more info refer to doc

Important: Do not forget to proper link celery app to your @shared_tasks. Celery application should be loaded before any task. For that reason define celery initialization in celery.py and then load it at myapp/__init__.py.

from __future__ import absolute_import

from .celery import app as celery_app  # noqa

Alternatively you can elaborate more on this link. It shows a good way of organizing your flask app together with celery app.

Solution 2

tasks.py

from myapp.app import celery

https://docs.python.org/3.5/tutorial/modules.html#packages

Solution 3

In the file that you initialise Celery, add this as the first line.

from __future__ import absolute_import

Explanation

from __future__ import absolute_import means that if you import celery, Python will look for a top-level celery module, rather than current_package.celery. This should possibly fix the error you are getting. But make sure celery is installed. It good to mention celery must be installed. This works for python version later than 2.6