scp: how to find out that copying was finished

Question

I'm using scp command to copy file from one Linux host to another. I run scp commend on host1 and copy file from host1 to host2. File is quite big and it takes for some time to copy it. On host2 file appears immediately as soon as copying was started. I can do everything with this file even if copying is still in progress.

Is there any reliable way to find out if copying was finished or not on host2?

Solution 1

Off the top of my head, you could do something like:

touch tinyfile
scp bigfile tinyfile user@host:

Then when tinyfile appears you know that the transfer of bigfile is complete.

As pointed out in the comments, this assumes that scp will copy the files one by one, in the order specified. If you don't trust it, you could do them one by one explicitly:

scp bigfile user@host:
scp tinyfile user@host:

The disadvantage of this approach is that you would potentially have to authenticate twice. If this were an issue you could use something like ssh-agent.

Solution 2

On sending side (host1) use script like this:

#!/bin/bash

echo 'starting transfer'
scp FILE USER@DST_SERVER:DST_PATH
OUT=$?
if [ $OUT = 0 ]; then
  echo 'transfer successful'
  touch successful
  scp successful USER@DST_SERVER:DST_PATH
else
  echo 'transfer faild'
fi

On receiving side (host2) make script like this:

#!/bin/bash 

SLEEP_TIME=30
MAX_CNT=10
CNT=0
while [[ ! -e successful && $CNT < $MAX_CNT ]]; do
    ((CNT++))

    sleep($SLEEP_TIME);
done; 

if [[ -e successful ]]; then
    echo 'successful'
    rm successful

    # do somethning with FILE
fi

With CNT and MAX_CNT you disable endless loop (in case file successful isn't transferred). Product MAX_CNT and SLEEP_TIME should be equal or greater expected transfer time. In my example expected transfer time is less than 300 seconds.

Solution 3

A checksum (md5sum, sha256sum ,sha512sum) of the local and remote files would tell you if they're identical.

For the situation where you don't have SSH access to the remote system - like an FTP server - you can download the file after it's uploaded and compare the checksums. I do this for files I send from production scripts at work. Below is a snippet from the script in which I do this.

MD5SRC=$(md5sum $LOCALFILE | cut -c 1-32)
MD5TESTFILE=$(mktemp -p /ramdisk)
curl \
    -o $MD5TESTFILE \
    -sS \
    -u $FTPUSER:$FTPPASS \
    ftp://$FTPHOST/$REMOTEFILE
MD5DST=$(md5sum $MD5TESTFILE | cut -c 1-32)
if [ "$MD5SRC" == "$MD5DST" ]
then
    echo "+Local and Remote files match!"
else 
    echo "-Local and Remote files don't match"
fi

Solution 4

After some investigation, and discussion of the problem on other forums I have found one more solution. Maybe it can help somebody.

There is a command "lsof". It lists open files. During copying the file will be opened, so the command

lsof | grep filename

will return non empty result.

So you might want to make a while loop to wait until lsof returns nothing and proceed with your task.

Example:

# provide your file name here
f=<nameOfYourFile>
lsofresult=`lsof | grep $f | wc -l`
while [ $lsofresult != 0 ]; do 
  echo still copying file $f...
  sleep 5
  lsofresult=`lsof | grep $f | wc -l`
done; echo copying file $f is finished: `ls $f`

Solution 5

if you use inotify-tools, then the solution will looks like this:

while ! inotifywait -e close $(dirname ${bigfile_fullname}) 2>/dev/null | \
    grep -Eo "CLOSE $(basename ${bigfile_fullname})$">/dev/null
do true
done
echo "File ${bigfile_fullname} closed"

View more solutions

Comments

[jfs]

unless scp decides that it can copy more than one file at a time.

[Tom Fenech]

@J.F.Sebastian I guess that's a possibility, I've edited my answer accordingly

[alvits]

lsof will query all process for open files. Why not directly check the file using fuser? fuser file will print the PIDs that have file opened and if the process is ssh then it is still being copied. Neither lsof nor fuser can be relied on because ssh will leave the file intact even when interrupted. The only reliable way is to get the checksum after the file has been closed. In your question you said "reliable way".

[v8-E]

Welcome to StackOverflow. Some explanation would greatly improve the quality of the answer and help other users who may encounter similar problems in the future Please take some time to read the help page stackoverflow.com/help/how-to-answer

[Nick M]

I'm upvoting this because it's a good answer and pretty much self-explanatory, thanks @molokovskikh