How can I capture output from LFTP? (Output not written to STDOUT or STDERR?)

Question

I would like get access to progress information from lftp. Currently, I'm using curl like so:

curl http://example.com/file -o file -L 2> download.log

This writes curl's progress information to the download.log file, which I can tail to get real-time progress.

But the same approach doesn't work with lftp, either with stdout or stderr. I end up with an empty download.log file, until the transfer is complete.

lftp -e 'get http://example.com/file;quit' 2> download.log
lftp -e 'get http://example.com/file;quit' 1> download.log

When I don't redirect output, I see progress on the screen. When I do redirect output, I stop seeing progress on the screen, but nothing shows up in download.log. After the file transfer is complete, I see the final result, like this - but nothing before:

97618627 bytes transferred in 104 seconds (913.1K/s)

Is lftp doing something unusual with its output - printing to screen without printing to stdout/stderr? Are there other ways of capturing screen output than redirecting stdout/stderr?

Solution 1

It sounds to me like it's buffering its output. You might try the unbuffer expect script (man page).

Solution 2

See for the "log:file/xfer" default setting by entering set command in lftp.

Possible output:

set log:file/xfer /home/USERNAME/.local/share/lftp/transfer_log

You can change this in different ways off course.

lftp -c 'open -e "set log:file/xfer /home/USERNAME/myown.log; mget *.csv; bye" -u uname,psswd 172.16.100.101' >> $log

Solution 3

Check the xfer domain variables :

set xfer:log 1

set xfer:eta-period 5 # every 5 seconds

set xfer:rate-period 20 # average rate

will put log transfer information into ~/.lftp/transfer_log

Not sure you can change the log file destination nevertheless

Comments

[benba]

from the manpage: xfer:log-file (path to file) the file to log transfers to. Default is ~/.local/share/lftp/transfer_log or ~/.lftp/transfer_log.

[juj]

Indead! This new setting xfer-log was added in version 4.2.0 by Timur Sufiev (2011-03-03)

[PiTheNumber]

Thanks, made my day! I just added unbuffer infront and it worked.