How to copy, execute and retrieve data from a remote computer?

Question

I am trying to run some calculations on a remote computer. Basically I want to copy some files to the computer, load a software, execute the calculation, then retrieve the data back. I have created a passwordless connection already for this. Below is my attempt:

#!/bin/bash
scp ./* username@host:/home/tmp
work=$pwd
ssh username@host
cd /home/tmp
module purge
module load gcc/4.6.0 icc/11.1.075  ifort/11.1.075 compilers-extra mpi-openmpi/1.4.3-icc-11.1 vasp
mpirun -np 20 vasp > out.vasp 2>&1 &
scp ./* username@headnode:$work

The script just copies the files and logs in to the remote computer, but no calculations are initiated.

Solution 1

I'm not really sure what your question is, but I have two remarks which are too long for a comment:

1. Your current script will stop once you log to host (at line 4) and will wait for you to do something interactively - only when you exit will the script proceed (which probably isn't what you want).

   Hence put anything that is to be executed on the remote server into a standalone shell script, scp it there with the rest of the data and run it with ssh username@host /path/to/my/script.

   Thus:

#!/bin/bash
# your local script
scp ./inputs* ./do_stuff.sh username@host:/home/tmp
ssh username@host /home/tmp/do_stuff.sh
scp username@host:/home/tmp/outputs* .

#!/bin/bash
# script for remote execution do_stuff.sh
cd /home/tmp
module purge
module load gcc/4.6.0 icc/11.1.075  ifort/11.1.075 compilers-extra mpi-openmpi/1.4.3-icc-11.1 vasp
mpirun -np 20 vasp > out.vasp 2>&1

replace inputs*/outputs* with whatever you need to copy to/from the cluster - you probably don't want to copy back the inputs. Note, that the script referenced to as do_stuff.sh has to be marked as executable. If even that doesn't work (because the filesystem you are copying it to is mounted with the noexec option, use ssh username@host bash /home/tmp/do_stuff.sh. Or you can just pipe it to the remote machine:

cat do_stuff.sh | ssh username@host bash -

2) If you are running this on headnode (which it seems you are), you can get rid of one direction of scp connection by running the scp on headnode again.

Solution 2

I would suggest you have your script in your home for remote execution and then call your script as,

ssh user@host 'bash -s' < local_script.sh

Basically, from your script with passwordless access, after ssh username@host, the script is not getting executed. This could be easily tested.

Testing

Have these 2 lines in a script and try running the script from your local machine.

ssh username@host
hostname

Normally, we would be expecting it would print the remote machine name since we have ssh'ed to it. But that is not the case. It just logs in to the remote machine and when you exit out of the remote machine, then only it will print host name (Be sure to verify what host name it is printing out, it would be printing out the local machine's host name).

Now, as I suggested, if you have your script in your local home and call it as,

ssh user@host 'bash -s' < local_script.sh

The output in this case is the remote host machine name (Be sure to remove the ssh line from the script as this is useless now).