Save a list of files over SFTP to a text file

Question

I have provided my own answer below.

We have a number of bash scripts that work fine using SFTP, what I would like to do is to simply re-direct an ls or dir of a folder to a file on our SFTP Server..

We can run this as a cron job or I can run it manually.. I can view the list of files from the remote server, but I want to generate the list of files from the remote server to the local server in a text file..

Here is what I have for the bash script so far.. The fourth echo listed below is the line in question.

#!/bin/bash
localpath=/home/localacct/somepath
remotelocation=/home/account/logs/archive
remotehost=' [email protected]'
    echo  > $localpath/list.sftp
    echo "cd $remotelocation " >> $localpath/list.sftp
    echo "dir *.* " >> $localpath/list.sftp
    echo "dir *.* > $localpath/dirlist.txt " >> $localpath/list.sftp
    echo "bye " >> $localpath/list.sftp
    sftp -b $localpath/list.sftp $remotehost
exit

Solution 1

There is another way which I wound up using.. Try using rsync.

We find the following:

• If you specify no local destination then a listing of the specified files on the remote server is provided.

The BASH script then becomes:

#!/bin/bash
localpath=/home/local-acct/path
remotelocation=/home/account/logs/archive
remotehost=' [email protected]'
rsync -avz $remotehost:/$remotelocation > $localpath/dirlist.txt
exit

and works great!

Solution 2

The sftp command is very limited. If you can't make it do what you want, you can use another approach, which is to mount the remote directory with the SSHFS filesystem. SSHFS uses SFTP as the transport protocol, so the server side just sees an SFTP client. You need to be able to use FUSE on the client side.

With SSHFS, you mount the remote directory onto an existing, empty directory and use ordinary commands.

mkdir remote
sshfs "$remotehost:$remotelocation" remote
cd remote
echo *.* >"$localpath/dirlist.txt"
fusermount -u remote
rmdir remote

Solution 3

This works by far the best:

echo 'ls' | sftp hostname

You can forward the output into a file by

echo 'ls' | sftp hostname > /tmp/mylist.txt

Solution 4

The dir command within the sftp client does not support redirection. Example below, showing how it does nothing.

sftp> pwd
Remote working directory: /var/tmp/foodir
sftp> lcd /var/tmp/foodir
sftp> dir *.*
foo.txt
sftp> dir *.* >dirlist.txt
foo.txt
sftp> dir
foo.txt
sftp>

Man page for sftp confirms.

ls [-1afhlnrSt] [path]

Display a remote directory listing of either path or the current directory if path is not specified. path may contain glob(3) characters and may match multiple files.

The following flags are recognized and alter the behaviour of ls accordingly:

-1

Produce single columnar output.
-a
List files beginning with a dot (‘.’).
-f
Do not sort the listing. The default sort order is lexicographical.
-h
When used with a long format option, use unit suffixes: Byte, Kilobyte, Megabyte, Gigabyte, Terabyte, Petabyte, and Exabyte in order to reduce the number of digits to four or fewer using powers of 2 for sizes (K=1024, M=1048576, etc.).
-l
Display additional details including permissions and ownership information.
-n
Produce a long listing with user and group information presented numerically.
-r
Reverse the sort order of the listing.
-S
Sort the listing by file size.
-t
Sort the listing by last modification time.

Solution 5

in case anyone finds rsync doesn't work (for reports.paypal.com in my case) I found the following script using expect works:

#!/usr/bin/expect
spawn sftp [email protected]:/ppreports/outgoing
expect "password:"
send "XXXXXXXXXX\n"
expect "sftp>"
log_file -noappend RemoteFileList.txt
send "ls -1\n"
expect "sftp>"
log_file
send "!sed -i '' '/ls -1/d' ./RemoteFileList.txt\n"
expect "sftp>"
send "!sed -i '' '/sftp>/d' ./RemoteFileList.txt\n"
expect "sftp>"
send "bye\n"
interact

This is on Mac. On other systems the sed lines might do without the first '' arguments.

View more solutions

Comments

[111---]

So what is this script not doing right?

[berto]

Just curious and probably stating the obvious: you must use SFTP, right? This will not work for your use case? ssh $remotehost ls > $localpath/list.sftp

[Арсений Черенков]

you should write you answer as separate answer and approve it.

[Stéphane Chazelas]

Note that rsync uses ssh (so assumes the remote user has shell access), not sftp (a subsystem of ssh that does not require the user having shell access) and assumes rsync is installed on the remote machine. So you might as well do ssh [email protected] ls /dir > file

[G-Man Says 'Reinstate Monica']

(1) I guess you’re trying to show variations on a theme, but this is a bit incoherent.  You demonstrate creating three files (one of them gets written twice).  Are they identical?  Is there any reason a user should prefer one of these commands over the others? (2) Why are you listing .. instead of remotelocation? (3) Have you tried this?  Does it really produce reasonable output?  I ask because I expect that echo "$a" would work much better than echo $a. (4) Does the name fct signify anything?

[dnlbrky]

Returning one file per row should make the output easier to handle: echo 'ls -1a' | sftp hostname