How to wget a file with correct name when redirected?

Question

So after some time of searching on Google and Super User (and scanning man pages) I was unable to find an answer to something that (I think) should be simple:

If you go here:

http://www.vim.org/scripts/script.php?script_id=2340

And try to download the theme:

http://www.vim.org/scripts/download_script.php?src_id=9750

Like so:

wget http://www.vim.org/scripts/download_script.php?src_id=9750

You’ll probably end up with a file called download_script.php?src_id=9750.

But I want it to be called molokai.vim, which is what would happen if I used a browser to download this file.

What options do I need to specify for wget for the desired effect?

I'd also be ok with a Curl equivalent command.

Solution 1

-O file
--output-document=file
   

The documents will not be written to the appropriate files, but all will be concatenated together and written to file. If - is used as file, documents will be printed to standard output, disabling link conversion. (Use ./- to print to a file literally named -.)

So,

wget -O somefile.extension 'http://www.vim.org/scripts/download_script.php?src_id=9750'

Or, you may be able to get wget to automatically use the filename proposed by the server using the --content-disposition option if supported by your version.

wget --content-disposition 'http://www.vim.org/scripts/download_script.php?src_id=9750'

Caveats as per the man page,

--content-disposition

If this is set to on, experimental (not fully-functional) support for "Content-Disposition" headers is enabled. This can currently result in extra round-trips to the server for a "HEAD" request, and is known to suffer from a few bugs, which is why it is not currently enabled by default.

This option is useful for some file-downloading CGI programs that use "Content-Disposition" headers to describe what the name of a downloaded file should be.

You can achieve the same automated behavior with curl, using,

curl -JLO 'http://www.vim.org/scripts/download_script.php?src_id=9750'

-O uses the remote name, and -J forces the -O to get that name from the content-disposition header rather than the URL, and -L follows redirects if needed.

Solution 2

With wget you can do this:

wget --trust-server-names <url> 

to save the file using the last file name the server gives you.

Solution 3

You could also use aria2c - it seems to work nicely with the Content-Disposition headers.

Solution 4

Worked by following:

curl -o molokai.vim http://www.vim.org/scripts/download_script.php?src_id=9750

wget -O somefile.extension http://www.vim.org/scripts/download_script.php?src_id=9750

(changed case to smaller i.e. (the wget -O) to (wget -o)

View more solutions

Comments

[audio.zoom]

I should've specified that this needs to be automatic

[EightBitTony]

As I understand it, wget relies on the web page to correctly inform it if the URL points to a file and it needs to be saved with something different, using the content disposition header, which wget may or may not fully support. So you can't do it automagically with wget. Curl may have more or less success depending on the web page.

[audio.zoom]

the point is to be able to script these things without complicated parsing, curl is fine but so far i haven't been able to collect decent options for that either

[EightBitTony]

So, running wget -S you can see the page does return a content disposition entry, Content-Disposition: attachment; filename=molokai.vim, which suggests it's not fixable. wget just isn't coping.

[EightBitTony]

Aha - I'm wrong, see updated answer. This is why stackexchange sites work well - everyone learns something, even people answering the questions!

[audio.zoom]

Nice one - easy tools are the best

[EightBitTony]

Added the curl parameters that make curl work too.

[Richard Campbell]

This worked for me for the stackexchange podcast, which had been bugging me for a while. Thanks.

[JamesTheAwesomeDude]

I wonder what the difference is between --trust-server-names and --content-disposition

[Asfand Qazi]

Looks like --trust-server-names follows redirects to a different file, and --content-disposition names the file with the name specified in the response header without any redirecting required.

[William Duclot]

Why isn't this default functionality?!?!

[Patrick Conheady]

@hopeseekr The server might be naughty and call the file .bashrc if you are not watching carefully. lists.gnu.org/archive/html/bug-wget/2012-04/msg00059.html

[avijit bhattacharjee]

So helpful. I was downloading something and its name was download without any extestion. So bothering.

[rocksNwaves]

--content-disposition worked well (at least it appears to be downloading the correct file as I type this), whereas curl -JLO did not do the trick. I'm new to using both tools, so I wonder why the disparity.

[vstepaniuk]

mg I love wget so much more than curl.

[vstepaniuk]

--content-disposition worked for Google Drive links for me, while --trust-server-names hasn't

[Stefan Schmidt]

You may want to consider using -R with curl to preserve the original time stamp of a file, resulting in curl -JLOR https://foobar.com/download.