Do we really need to check the integrity of downloaded files?

Question

I downloaded a huge amount of files, but discovered the use of md5 and sha as integrity checkers quite recently. From then I always prefer to check it for big downloaded files, even if I never found them to be corrupted.

Do we really need to check the integrity of downloaded files?

Pick as example a Linux distribution that I have just downloaded, which is 1GB, if you want.

Thank you

Solution 1

It depends on a few factors.

1. Do you have a stable internet connection?
   If you have a stable internet connection, you do not need to check the integrety of the file as it will most likely be correct. I never check the hash and I also never have had corrupt files. Or maybe one time when the remote server disconnected.

2. Do you want to verify the file based on security reasons?
   If you are concerned about the safety of the file you're downloading, you can use the MD5 hash to verify that the file was not somehow altered. You download a file and if the MD5 hash doesn't match, it means the file on the server is different than the MD5 hash, and somehow something is wrong. This would only be valid if you don't trust the server you're downloading from, but usually if someone provides a hash, they usually also try their best to keep things updated. But if their site got hacked, and you did check the MD5 hash, then you got the little bonus.

Overall, these 2 will give a no to most people. If its a no to you, thats entirely up to you of course.

Solution 2

The answer and choice one makes is going to be based on his/her risk tolerance and considerations of time and effort in verification.

Checking MD5/SHA1 hashes is a good first step and you should do it when you have time. However, you must consider your ability to trust the hash provided. For example, if the the author's website with the hash is hacked, then the attacker can change the hash, so you would not know. If the hash you calculate is not the same as the hash provided, you know something is up. However, just because the hashes match does not guarantee the file is good.

A better alternative for a software author to provide integrity and authenticity is through digitally signing the files being distributed. This attaches the authenticity information to the file and does not rely on trusting some website. If an author digitally signs the file, the only way for this to be faked is a compromised certificate authority or if the developer's signing key was stolen. Both of these cases are far less likely than a website on the Internet being hacked.

Ultimately, you must do your own due diligence to determine if you want to trust something and then take countermeasures (run in a sandbox, a virtual machine, etc.) to mitigate any unknown factors or miscalculations you made when deciding whether or not to trust.

Solution 3

For security reasons, YES. Consider that a Tor exit node was found to be patching binaries during download, then remember that your ISP may or may not have the slightest morals, and that they are in complete control of your internet connection.

Solution 4

For files where integrity is critical, yes

I often find it necessary when I am doing something where it is critical that the file has a high integrity.

One example is flashing a router with OpenWRT. If the file is corrupt, then that router would be bricked, and then I would either have to either:

• Replace it (expensive)
• Fix it (time consuming. Especially if I need to solder a serial cable/JTAG)

Both of these are inconvenient, compared to the simplicity of checking a hash. Therefore I would strongly recommend doing it for critical files.

View more solutions

Comments

[Dan]

Many downloads are http downloads and not few of the big ones stop prematurely due to server time limits or connectivity issues, without giving any error. Torrents and ftp on the other hand should make it clear if the file is complete.

[TheUser1024]

The point of comparing the hash can rather be "has someone altered the file (maliciously)" than "has the file been downloaded correctly". One prime example is to download a windows ISO from a non-official/not trustworthy source and check the integrity of the file by comparing the MD5 hash with the one Microsoft published on MSDN.

[Eric G]

@TheUser1024 this assume that your two sources are the same, the MSDN version may be different.

[Eric G]

This question may get more responses on security.SE

[maxpesa]

It was actually more on integrity than security, since I only download from trusted sources, even if it is an important point in file download

[BrianH]

Back in the "good old days" when downloading files from Usenet, especially multiple GB ones that were split into many smaller pieces, it was nearly a necessity to use checksums and often redownload individual parts. Parity files were even necessary and greatly welcomed, because the nature of the exchange was so likely to produce errors in the files that without such measures you'd never be able to get something useful downloaded. Now a days this isn't nearly so bad - you can download 100GB and not have a bad bit on a broadband connection.

[user55325]

Even with a stable internet connection I still check the hash before and after I burn an image. It only takes a minute and it'll ensure that I don't run into errors during the installation (this is rare, but it has happened to me. Maybe 1% of the time.)

[ChrisInEdmonton]

A stable internet connection is not sufficient. You also have to be sure your RAM and storage drive do not corrupt the files. But yeah, pretty unlikely. If your machine isn't BSOD'ing, chances are you would only care about the security.

[maxpesa]

Your complain on ISP is on the fact that they can change my connection speed and priority and other things that can corrupt files?

[Ramhound]

@maxpesa - They can modify the stream if they want. The file won't be corrupt just modified.

[Michael Kohne]

@maxpesa - I was pointing out that your ISP is in the same position that the Tor exit node in the linked article was - it has the ability to control what comes over your wires. If they wanted to, they are in position to modify the binaries you download on-the-fly.

[kapex]

If the attacker can modify the binary, the attacker can most likley also modify the hash. That would require a more targeted attack than patching all binaries though, but is still very possible. While checking the hash can avoid damage, there is no guarantee. If you care about security, the first step is to only download over HTTPS - especially when you are using proxies like with tor!

[Eric G]

I would consider that the intermediary may also change the source of the hash against which to validate. Encryption/HTTPS may counteract some of these scenarios to prevent such MitM attacks.

[Johnny]

To add to @kapep's comment, it's even better to validate using cryptographical signatures (using keys distributed through a secure channel). That way an attacker would have to compromise both the keystore and the repository in order to alter the binary.

[Johnny]

A file checksum is not a security check since an attacker that can modify the file could likely also modify the checksum.

[Cole Tobin]

If I were to hack a site and replace the file with a malicious one, I think I'd be smart enough to know to change the hash listed.

[Perkins]

MD5 is no longer sufficient to verify that the contents of a file have not been intentionally altered. It's only good for unintentional corruption. The odds of an attacker compromising both vary. The storage for the hash, in many cases, is not accessed through the same channels as the bulk file storage for a large download. For security, cryptographic signatures are better than simple hashes, especially if you don't have a secure channel for distributing the hashes.

[tvt173]

I think the idea of hashes was when the file supplied was hosted on a different site to the hash. Eg a developer's site with a hash links to Sourceforge or somewhere.