How do I setup curl / wget with FTP proxy?

Question

I'm having a good deal of trouble getting curl or wget to download files from FTP servers through an FTP proxy. I have setup GUI FTP programs to work...

gFTP works with these settings:

I get output like below when just using curl/wget with the ftp_proxy variable set to the proxy hostname:

# curl -v ftp://ftp.astron.com/pub/file/file-5.05.tar.gz
* About to connect() to proxy blah port 21 (#0)
*   Trying blah... connected
* Connected to blah (blah) port 21 (#0)
> GET ftp://ftp.astron.com/pub/file/file-5.05.tar.gz HTTP/1.1
> User-Agent: curl/7.19.0 (x86_64-suse-linux-gnu) libcurl/7.19.0 OpenSSL/0.9.8h zlib/1.2.3 libidn/1.10
> Host: ftp.astron.com:21
> Pragma: no-cache
> Accept: */*
> Proxy-Connection: Keep-Alive
>
220-
220-Enter an Internet ftp address at the Name prompt.
220 Type help for usage information.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.
500 Syntax error, command unrecognized.

Is there some way I can configure the shell environment and/or curl/wget with the same settings as gFTP for passing through an FTP proxy?

Solution

The problem here seems to be the type of proxy used.

The proxy you are using is an FTP proxy. Wget`and Curl both use HTTP proxies.

Comments

[entropo]

Both curl and wget support FTP proxies... wget: gnu.org/software/wget/manual/wget.html#Proxies ... curl: curl.haxx.se/docs/manual.html

[FJ de Brienne]

Ah no - they support proxies for FTP transfers using HTTP proxies - not FTP native proxies. The protocols are different. The "GET blah.blah HTTP/1.1" in your debug output is a dead giveaway - that's an HTTP proxy request that is being generated.

[entropo]

Hmm, okay. Are there any command line tools which DO support native FTP proxying?

[FJ de Brienne]

You may not need them - the way the FTP proxy works you may be able to slip the details into the URL... f t p://[email protected]@ftp.proxy.com/path/to/file without any proxy set might possibly work. It's worth a try anyway

[entropo]

Aha! Victory, thank you. That's stupidly painful, but here's my regex for reworking the URLs: perl -pe 's/(ftp\:\/\/)(\S+?)\/(\S+)$/$1ftp%40$2\@proxy.blah.com\/$3/‌​'

[entropo]

For further reference, the "Types of firewalls" list here is useful: linuxnet.ch/groups/linuxnet/wiki/a6fa7/FTP_via_Proxy.html

[entropo]

Also, I found there is a feature request filed for wget to support FTP proxy auth: savannah.gnu.org/bugs/index.php?21439

[GuruM]

curl is usable with ftp proxy as per the curl manual. The following command worked for me with pure ftp server with a ftp proxy in between. curl -u "[email protected] :anonymous" ftp://myproxy:21/path/to/server-ftp-file

[GuruM]

Extract from the manual: Most FTP proxy servers are set up to appear as a normal FTP server from the client's perspective, with special commands to select the remote FTP server. curl supports the -u, -Q and --ftp-account options that can be used to set up transfers through many FTP proxies. For example, a file can be uploaded to a remote FTP server using a Blue Coat FTP proxy with the options: curl -u "[email protected] Proxy-Username:Remote-Pass" --ftp-account Proxy-Password --upload-file local-file ftp://my-ftp.proxy.server:21/remote/upload/path

[GuruM]

Using other options of curl I was still seeing the GET ... HTTP 1.1 in the verbose output. Even forcing curl with --proto="=ftp" didn't work. But the above command did. Hope this helps someone else.