Recursive FTP directory listing in shell/bash with a single session (using cURL or ftp)

Question

I am writing a little shellscript that needs to go through all folders and files on an ftp server (recursively). So far everything works fine using cURL - but it's pretty slow, becuase cURL starts a new session for every command. So for 500 directories, cURL preforms 500 logins.

Does anybody know, whether I can stay logged in using cURL (this would be my favourite solution) or how I can use ftp with only one session in a shell script?

I know how to execute a set of ftp commands and retrieve the response, but for the recursive listing, it has to be a little more dynamic...

Thanks for your help!

Solution 1

The command is actually ncftpls -R. It will recursively list all the files in a ftp folder.

Solution 2

Just to summarize what others have said so far. If you are trying to write a portable shell script which works as batch file, then you need to use the lftp solution since some FTP server may not implement ls -R. Simply replace 123.456.789.100 with the actual IP adress of the ftp server in the following examples:

$ lftp -c "open 123.456.789.100 && find -l && exit" > listing.txt

See the man page of lftp, go to the find section:

List files in the directory (current directory by default) recursively. This can help with servers lacking ls -R support. You can redirect output of this command.

However if you have a way to figure out whether or not the remote ftp server implements proper support for ls -lR, then a much better (=faster) solution will be:

$ echo ls -lR | ftp 123.456.789.100 > listing.txt

Just for reference if I execute the first command (lftp+find) it takes 0m55.384s to retrieve the full listing, while if I execute the second one (ftp+ls-R), it takes 0m3.225s.

Solution 3

If it's possible, try usign lftp script:

# lftp script "myscript.lftp"
open your-ftp-host
user username password
cd directory_with_subdirs_u_want_to_list
find
exit

Next thing u need is bash script to run this lftp command and write it to file:

#!/bin/bash
lftp -f myscript.lftp > myOutputFile

myOutputFile now contains the full dump of directories.