Loading shared library in open-mpi/ mpi-run

Question

I'm trying to run my program using torque scheduler using mpi run. Though in my pbs file I load all the library by

export LD_LIBRARY_PATH=/path/to/library

yet it gives error i.e.

error while loading shared libraries: libarmadillo.so.3: 
cannot open shared object file: No such file or directory. 

I guess error lies in variable LD_LIBRARY_PATH not set in all the nodes. How would I make it work?

Solution 1

LD_LIBRARY_PATH is not exported automatically to MPI processes, spawned by mpirun. You should use

mpirun -x LD_LIBRARY_PATH ...

to push the value of LD_LIBRARY_PATH. Also make sure that the specified path exists on all nodes in the cluster and that libarmadillo.so.3 is available everywhere.

Solution 2

On some systems, your environment isn't always propagated via mpirun. You should set all those variables in your .bashrc file.

Comments

[chrisaycock]

Could you post your PBS script here?

[Jeff Squyres]

If Open MPI was compiled with native Torque/TM support, the LD_LIBRARY_PATH (and the rest of the environment) should be propagated out to all the nodes in the allocation automatically. If Open MPI wasn't built with the native Torque support, then I'm guessing you're using rsh/ssh to launch, in which case the "-x LD_LIBRARY_PATH" clause is necessary. I'd actually advise re-building OMPI with Torque/TM support; it's better than rsh/ssh support for multiple reasons.

[Greg K.]

THANK YOU. I WAS DRIVING MAD WITH THIS SHIT.... Exporting the LD_LIBRARY_PATH is supposed to let child processes use those variables, but it turns out it doesn't.

[Hristo Iliev]

@GregK. exporting only works for processes launched on the same node where mpiexec runs as those are child processes of the latter. Processes launched on remote nodes are children of the respective ORTE shepherd daemon and don't directly inherit the environment of mpiexec but only the environment that gets forwarded to and additionally set by the daemon. (I might be wrong; Jeff Squyres knows better)