Bash get MD5 of online file

Question

I need to get the MD5 hash of an online file, and then compare it to a file on the local machine.

How can I do this in bash?

Solution 1

You can use curl to fetch the online file:

curl -sL http://www.your.fi/le | md5sum | cut -d ' ' -f 1

To compare against another one, store it in a variable and then proceed:

online_md5="$(curl -sL http://www.your.fi/le | md5sum | cut -d ' ' -f 1)"
local_md5="$(md5sum "$file" | cut -d ' ' -f 1)"

if [ "$online_md5" = "$local_md5" ]; then
    echo "hurray, they are equal!"
fi

Solution 2

You could also just do it directly. Use wget or curl to print the remote file's contents and also print the contents of the local file. Pass both to md5sum and compare the output:

$ md5sum <(wget  http://www.exacmple.com/file -O- 2>/dev/null) <(cat localfile) 
733f328d8cff7dd89970ec34a70aa14f  /dev/fd/63
733f328d8cff7dd89970ec34a70aa14f  /dev/fd/62

The first line will be the remote file's md5sum and the second is the local's.

Solution 3

wget can download to standard output with -O-.

 wget http://example.com/some-file.html -O- \
     | md5sum \
     | cut -f1 -d' ' \
     | diff - <(md5sum local-file.html | cut -f1 -d' ')

md5sum appends the file name after the MD5, you can remove it with cut.

Solution 4

 wget -q -O- http://example.com/your_file | md5sum | sed 's:-$:local_file:' | md5sum -c

Replace http://example.com/your_file with the URL of your online file and local_file with the name of your local file

View more solutions

Comments

[Finlay Roelofs]

Could you please explain what the diff - <(md5sum local-file.html | cut -f1 -d' ') does?

[Finlay Roelofs]

Could you please explain what the sed does?

[Christopher B. Adkins]

The output of md5sum is a line containing the checksum and the file name. md5sum -c checks that file name for the checksum. the sed command replaces the - that md5sum uses for stdin with the name of the local file so the md5sum -c at the end verifies that the local file's checksum is the one of the online file.

[choroba]

@FinlayRoelofs: it runs diff on the pipe output and output of the command in <(...) process substitution, i.e. it compares the MD5 sums of the downloaded file and the local file as requested.

[Rafael]

are these expected to be the same? for example running: md5sum <(wget http://stackoverflow.com/opensearch.xml -O- 2>/dev/null) <(cat P:/stack.xml) gives me two different checksums...is there a certain way to download the file?

[terdon]

@R.M. what file? If you get different checksums then the files are different. What is P:/stack.xml? Does it maybe have a final newline? Did you save it in a text editor? That would have added a newline.

[Rafael]

i downloaded it using download.file(..., mode = 'wb') in R.

[terdon]

@R.M. then it almost certainly added a newline (\n or, if on Windows as your path suggests, \r\n) to the end of the file. So the content isn't exactly the same because of that final newline.

[Rafael]

Hope you don't mind me linking to my question, but I don't think the \r\n was the issue--I tried truncating the file by 1 and 2 bytes and still got different checksums than what wget returns. Here's the example: stackoverflow.com/questions/56602271/…

[terdon]

@R.M. either way, if the checksums are different, the data are different. So you will need to figure out why you are getting different data.