How can I get the latest files from a Unix server to another Unix server?

Question

How can I get the only the latest files from a Unix server to another Unix server?

I have connected via SFTP to server 2 and trying to apply a for loop to check the time stamp, which is not working:

sftp $SERVER2  << !EOF
cd $Server2_FILE_LOCATION
echo Pwd File location: $pwd

LastUpdatedTS=$(grep "value of TimeStamp is" /root/airtelSnD/BoTreeScript/BotreeLastFileTS.txt | cut -d'=' -f2)

echo 1

for file in mad_*.*
do
CurrentFileTS=$(stat -c %Y $file |awk '{print  strftime( "%Y%m%d%H%M%S", $1 )}')

echo 2

echo TS of last updated file is : $LastUpdatedTS
echo value CurrentFileTS is $CurrentFileTS

echo 3

if [[ $CurrentFileTS -gt $LastUpdatedTS ]]
 then
    echo if......
    mget  $file $DESTINATION_SERVER
    echo value of TimeStamp is=$CurrentFileTS > $LASTFILE_TS    
else
    echo else...       
 fi
done

quit
!EOF

Solution

Have you tried rsync?

rsync is an open source utility that provides fast incremental file transfer.

It will only transfer changed/new files, and only deltas for changed files. It also has various other nifty features.

This doesn't really address any issues with your code, but I think rsync will be easier than trying to reinvent the wheel with SFTP.

Comments

[Scott - Слава Україні]

I agree with the posted answers that you should use a tool that is designed to do this task. But if you want to know how to get scripts like this to work, try putting a backslash before all dollar sign symbols except for the ones that are using a value from the sftp environment. (It looks like $Server2_FILE_LOCATION and $DESTINATION_SERVER are the only ones to which that applies.)

[PetaspeedBeaver]

bolD's answer to the following question might interest you: scp-without-replacing-existing-files-in-the-destination

[Maddy]

Nope,as per my requirement I dont have to use any tool. I need to do it through scripts only.