Why can't Python find shared objects that are in directories in sys.path?
Solution 1
sys.path
is only searched for Python modules. For dynamic linked libraries, the paths searched must be in LD_LIBRARY_PATH
. Check if your LD_LIBRARY_PATH
includes /usr/local/lib
, and if it doesn't, add it and try again.
Some more information (source):
In Linux, the environment variable LD_LIBRARY_PATH is a colon-separated set of directories where libraries should be searched for first, before the standard set of directories; this is useful when debugging a new library or using a nonstandard library for special purposes. The environment variable LD_PRELOAD lists shared libraries with functions that override the standard set, just as /etc/ld.so.preload does. These are implemented by the loader /lib/ld-linux.so. I should note that, while LD_LIBRARY_PATH works on many Unix-like systems, it doesn't work on all; for example, this functionality is available on HP-UX but as the environment variable SHLIB_PATH, and on AIX this functionality is through the variable LIBPATH (with the same syntax, a colon-separated list).
Update: to set LD_LIBRARY_PATH
, use one of the following, ideally in your ~/.bashrc
or equivalent file:
export LD_LIBRARY_PATH=/usr/local/lib
or
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
Use the first form if it's empty (equivalent to the empty string, or not present at all), and the second form if it isn't. Note the use of export.
Solution 2
Ensure your libcurl.so
module is in the system library path, which is distinct and separate from the python library path.
A "quick fix" is to add this path to a LD_LIBRARY_PATH
variable. However, setting that system wide (or even account wide) is a BAD IDEA, as it is possible to set it in such a way that some programs will find a library it shouldn't, or even worse, open up security holes.
If your "locally installed libraries" are installed in, for example, /usr/local/lib
, add this directory to /etc/ld.so.conf
(it's a text file) and run ldconfig
The command will run a caching utility, but will also create all the necessary "symbolic links" required for the loader system to function. It is surprising that the make install
for libcurl did not do this already, but it's possible it could not if /usr/local/lib
is not in /etc/ld.so.conf
already.
PS: it's possible that your /etc/ld.so.conf
contains nothing but include ld.so.conf.d/*.conf
. You can still add a directory path after it, or just create a new file inside the directory it's being included from. Dont forget to run ldconfig
after it.
Be careful. Getting this wrong can screw up your system.
Additionally: make sure your python module is compiled against THAT version of libcurl. If you just copied some files over from another system, this wont always work. If in doubt, compile your modules on the system you intend to run them on.
Solution 3
You can also set LD_RUN_PATH to /usr/local/lib in your user environment when you compile pycurl in the first place. This will embed /usr/local/lib in the RPATH attribute of the C extension module .so so that it automatically knows where to find the library at run time without having to have LD_LIBRARY_PATH set at run time.
Solution 4
Had the exact same issue. I installed curl 7.19 to /opt/curl/ to make sure that I would not affect current curl on our production servers. Once I linked libcurl.so.4 to /usr/lib:
sudo ln -s /opt/curl/lib/libcurl.so /usr/lib/libcurl.so.4
I still got the same error! Durf.
But running ldconfig make the linkage for me and that worked. No need to set the LD_RUN_PATH or LD_LIBRARY_PATH at all. Just needed to run ldconfig.
Solution 5
As a supplement to above answers - I'm just bumping into a similar problem, and working completely of the default installed python.
When I call the example of the shared object library I'm looking for with LD_LIBRARY_PATH
, I get something like this:
$ LD_LIBRARY_PATH=/path/to/mysodir:$LD_LIBRARY_PATH python example-so-user.py
python: can't open file 'example-so-user.py': [Errno 2] No such file or directory
Notably, it doesn't even complain about the import - it complains about the source file!
But if I force loading of the object using LD_PRELOAD
:
$ LD_PRELOAD=/path/to/mysodir/mypyobj.so python example-so-user.py
python: error while loading shared libraries: libtiff.so.5: cannot open shared object file: No such file or directory
... I immediately get a more meaningful error message - about a missing dependency!
Just thought I'd jot this down here - cheers!
BR123
Updated on July 08, 2022Comments
-
BR123 almost 2 years
I'm trying to import
pycurl
:$ python -c "import pycurl" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: libcurl.so.4: cannot open shared object file: No such file or directory
Now,
libcurl.so.4
is in/usr/local/lib
. As you can see, this is insys.path
:$ python -c "import sys; print(sys.path)" ['', '/usr/local/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg', '/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', '/usr/local/lib/python2.5/plat-linux2', '/usr/local/lib/python2.5/lib-tk', '/usr/local/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/sitepackages', '/usr/local/lib', '/usr/local/lib/python2.5/site-packages']
Any help will be greatly appreciated.
-
Vinay Sajip almost 15 yearsSee my updated answer, in case you didn't set
LD_LIBRARY_PATH
correctly (I thought your comment had a missing colon). -
Charles Duffy almost 15 yearsIs there a broken symlink somewhere named libcurl.so.4? It looks to me like it's finding the file but unable to open it. If all else fails, strace the interpreter and look for the failing call.
-
-
Admin almost 15 yearsThanks. My LD_LIBRARY_PATH was not set, so: $ LD_LIBRARY_PATH=/usr/local/lib $LD_LIBRARY_PATH /usr/local/lib But I still get the same error: $ python -c "import pycurl" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: libcurl.so.4: cannot open shared object file: No such file or directory
-
Admin almost 15 yearsThank you - this worked. I wonder why my previously attempted "quick fix" changing the LD_LIBRARY_PATH variable did not.
-
Ch'marr over 14 yearsDepends on a lot of factors. Here's one possibility: your code was being run from apache or cron. Those programs typically "clean out" the environment, so you have to do extra stuff to get environment variables in. For example, "SetEnv" in apache, or setting the variable right in the crontab file for cron. The possibilities for mistakes is endless!
-
sprajagopal almost 11 yearsWhat if I don't have sudo privilege? I cannot run ldconfig? Is there any way to clear the above error then?
-
kynan almost 11 yearsAlternatively, use
python setup.py build_ext --rpath=/usr/local/lib
when building the extension module to bake in the rpath -
José Ricardo almost 11 yearsI had also to give permissions to my user to read the library after setting the LD_LIBRARY_PATH variable. Now it finally works.
-
MadScientist almost 11 years@SPRajagopal: if you don't have privileges to modify the system attributes you have to use the
LD_LIBRARY_PATH
environment variable method described above. If you don't want to set it in your~/.bashrc
(adding that setting is not a good idea IMO) you can write a shell script that sets this variable then runs python, then call that script. -
David Knipe about 7 yearsAre you sure that's not a new error that's happening before OP's error?