Error importing Google Cloud Bigquery api module in python app

Question

I am trying to import bigquery into my python app from google.cloud import bigqueryand run it locally with dev_appserver.py, but I receive an error:

File "/Volumes/Budhi/Users/anjas/Projects/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/Volumes/Budhi/Users/anjas/Projects/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/Volumes/Budhi/Users/anjas/Projects/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/Volumes/Budhi/Users/anjas/Projects/wordworks/urlworker/main.py", line 9, in <module>
from google.cloud import bigquery
File "/Volumes/Budhi/Users/anjas/Projects/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 999, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named google.cloud.bigquery

I have installed bigquery lib with pip:

pip install --upgrade google-cloud-bigquery

Also I've tried installing it as 3d party library into lib directory with no result.

Though it works when I try importing bigquery lib from python shell:

Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more  information.
>>> from google.cloud import bigquery
>>> 

Update:

It seems that "google" module installed to project/lib/ folder is clashing with "google" module in google-cloud-sdk/platform/google_appengine.

When trying to do

from google.cloud import bigquery

python looks for the module inside google-cloud-sdk/platform/google_appengine/google not inside project/lib/google/cloud

Any ideas?

Solution 1

I experienced exactly the same error and fixed this by deleting google folder in appengine sdk or you can just try to copy that to your lib folder . This will fix the problem on local development environment but you need to revert this when deploying your service on production.

I could not find a workaround to keep both google-cloud-sdk/platform/google_appengine/google and project/lib/google/cloud. You need to delete one of them

Solution 2

Did you add the lib folder in the appengine_config.py?

from google.appengine.ext import vendor
vendor.add('lib')

Solution 3

I had the same problem. If you're running this on your local environment, you may want to try adding the lib folder location into your $PYTHONPATH environment variable or adding below code lines before you import the bigquery lib which inserting the lib folder location into the a list of strings that specifies the search path for modules.

import sys    
sys.path.insert(0, 'lib')

I believe this works because now python files don't need to be inside of a package.

Comments

[Andrei Ivasiuc]

Yes I did. I've came across some other thing though, the package installed lib/google had no init.py inside, so it was not recognized as package. Adding init.py solves nothing. Seems like this SDK is meant to be installed globally. But I have installed it globally and yet no result.

[Martijn Pieters]

vendor.add('lib') is basically a wrapper around sys.path.insert().