How do I properly set wget to download only new files?

Question

Let's say there's an url, let's call it http://www.some-url.com/folder/

This location has directory listing enabled, therefore I can do this:

wget -r -np http://www.some-url.com/folder/

To download all its contents with all the files and subfolders and their files.

Now, what should I do if I want to repeat this process again, a month later, and I don't want to download everything again, only add new/changed files?

Solution 1

did you read this ? http://www.editcorp.com/Personal/Lars_Appel/wget/wget_5.html

"Several days later, you would like Wget to check if the remote file has changed, and download it if it has."

wget -N http://www.some-url.com/folder/

Try this.

Solution 2

wget -c 

also continues partial files caused when the download has issues.....
Better yet,

 wget -c -N 

seems to do both at the same time.

Comments

[Matt Williamson]

I found with my version 1.11.4 Red Hat modified, these flags are not compatible and -c overrides -N.

[unibasil]

But -N is useless if retrieved file has not Last-Modified header.

[mchawre]

-c option of wget helped mw the sync partially downloaded file from url. Its really a good option to use if faced any network problems while downloading.

[jarno]

@unibasil but how do you know, if the retrieved file has Last-Modified header?

[Adrian]

I can confirm Matt Williamson's observation, using the -c flag nullifies the effect of -N, meaning newer files will not get downloaded.