Installing tensorflow with GPU support. libcublas.so.9.0 missing after installing GPU prerequisites

Question

I am trying to install Tensorflow with GPU support on Ubuntu 16.04 64x for an conda environment with Python 3.6.

I tried installing all the GPU requirements and then running pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.10.0-cp36-cp36m-linux_x86_64.whl from my conda environment.

However when I open a Python terminal and try import tensorflow as tf I get a ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory.

This is the state of the requirements:

• NVIDIA driver version: 384.130 (this is the output of nvidia-smi)
• CUDA compiler driver: release 7.5, V7.5.17 (this the output of nvcc -v)
• CUDA: Version 9.2.148 (this is the output of cat /usr/local/cuda/version.txt). I am really confused because I read somewhere else that the CUDA version and the nvcc version should match.
• cuDNN: I think I have installed it? I downloaded the .deb package and I did sudo dpkg -i /path/to/deb/file followed by sudo apt-get install -f. But the internet tells me that executing cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2 should give me the cuDNN version and it complains that the file does not exist.
• CUPTI: I ran export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64 as indicated in the tensorflow requirements guide.

What should I try now?

---

Full error trace:

>>> import tensorflow
Traceback (most recent call last):
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/__init__.py", line 22, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/jsevillamol/anaconda3/envs/ctlearn/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

Solution 1

Cuda 9.0 can be installed with codes on the following tutorial

https://www.tensorflow.org/install/gpu

# Add NVIDIA package repository
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
sudo apt install ./cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt update

# Install CUDA and tools. Include optional NCCL 2.x
sudo apt install cuda9.0 cuda-cublas-9-0 cuda-cufft-9-0 cuda-curand-9-0 \
    cuda-cusolver-9-0 cuda-cusparse-9-0 libcudnn7=7.2.1.38-1+cuda9.0 \
    libnccl2=2.2.13-1+cuda9.0 cuda-command-line-tools-9-0

# Optional: Install the TensorRT runtime (must be after CUDA install)
sudo apt update
sudo apt install libnvinfer4=4.1.2-1+cuda9.0

Solution 2

Fixed! Turns out that the TF distro installed by default does not support CUDA 9.2. I downgraded to CUDA 9.0 and now it is tentatively working.