OSX: Generate MD5 checksum recursively in a textfile containing files with corresponding checksum

Question

In a directory withmultiple subdirectories but only one folder deep containing tiff-files I'd like to generate a md5 checksum that writes the filename with the corresponding checksum into a textfile.

For example in directory TIFF I have 2 subdirectories:

TIFF
  |- b0125TIFF
        |- b_0000_001.tif
        |- b_0000_002.tif
        |- b_0000_003.tif
        |- b_0000_004.tif
  |- c0126TIFF
        |- c_0000_001.tif
        |- c_0000_002.tif
        |- c_0000_003.tif
        |- c_0000_004.tif

My expected textfile (checksum should be of course different):

** foo.md5:
188be1dbd4f6bcfdef8d25639473e6ec *b0125TIFF/b_0000_001.tif
188be1dbd4f6bcfdef8d25639473e6ec *b0125TIFF/b_0000_002.tif
188be1dbd4f6bcfdef8d25639473e6ec *b0125TIFF/b_0000_003.tif
188be1dbd4f6bcfdef8d25639473e6ec *b0125TIFF/b_0000_004.tif
188be1dbd4f6bcfdef8d25639473e6ec *c0126TIFF/c_0000_001.tif
188be1dbd4f6bcfdef8d25639473e6ec *c0126TIFF/c_0000_002.tif
188be1dbd4f6bcfdef8d25639473e6ec *c0126TIFF/c_0000_003.tif
188be1dbd4f6bcfdef8d25639473e6ec *c0126TIFF/c_0000_004.tif

How can I achieve that?

I know that this generates the checksum recursively in one directory:

find -s . -type f -exec md5 -q {} \; | md5

Solution

You don't want to pass the output of the find and md5 through md5, that would just give you an MD5 checksum of a lot of MD5 checksums...

---

$ find TIFF -type f -name '*.tif' -exec md5 {} ';' >md5.txt
$ cat md5.txt
MD5 (TIFF/b0125TIFF/file-1.tif) = d41d8cd98f00b204e9800998ecf8427e
MD5 (TIFF/b0125TIFF/file-2.tif) = d41d8cd98f00b204e9800998ecf8427e
MD5 (TIFF/b0125TIFF/file-3.tif) = d41d8cd98f00b204e9800998ecf8427e
MD5 (TIFF/c0126TIFF/file-1.tif) = d41d8cd98f00b204e9800998ecf8427e
MD5 (TIFF/c0126TIFF/file-2.tif) = d41d8cd98f00b204e9800998ecf8427e
MD5 (TIFF/c0126TIFF/file-3.tif) = d41d8cd98f00b204e9800998ecf8427e

The md5 implementation on macOS does not support verifying checksums with md5 -c unfortunately, but the shasum utility does:

$ find TIFF -type f -name '*.tif' -exec shasum {} ';' >sums.txt
$ cat sums.txt
da39a3ee5e6b4b0d3255bfef95601890afd80709  TIFF/b0125TIFF/file-1.tif
da39a3ee5e6b4b0d3255bfef95601890afd80709  TIFF/b0125TIFF/file-2.tif
da39a3ee5e6b4b0d3255bfef95601890afd80709  TIFF/b0125TIFF/file-3.tif
da39a3ee5e6b4b0d3255bfef95601890afd80709  TIFF/c0126TIFF/file-1.tif
da39a3ee5e6b4b0d3255bfef95601890afd80709  TIFF/c0126TIFF/file-2.tif
da39a3ee5e6b4b0d3255bfef95601890afd80709  TIFF/c0126TIFF/file-3.tif

$ shasum -c sums.txt
TIFF/b0125TIFF/file-1.tif: OK
TIFF/b0125TIFF/file-2.tif: OK
TIFF/b0125TIFF/file-3.tif: OK
TIFF/c0126TIFF/file-1.tif: OK
TIFF/c0126TIFF/file-2.tif: OK
TIFF/c0126TIFF/file-3.tif: OK

shasum calculates the SHA1 hash of a file by default.

Comments

[Admin]

This is what md5 does by default if you leave the -q flag out and give it multiple files... Are you not happy with the default output of md5?

[Admin]

Oh, I didn't know. I started the command find -s . -type f -exec md5 -q {} \; | md5 and will check the result but it takes several hours (have 3TB on tiff files). Will close the question afterwards.

[Admin]

The creation of md5 checksums work great, thank you very much! But when I try $ md5 -c md5.txt I get md5: illegal option -- c usage: md5 [-pqrtx] [-s string] [files ...]

[Kusalananda]

@EmilyChoi Ah, the md5 on macOS does not have the -c flag. Then you may either use shasum instead or install GNU coreutils and use md5sum from there. I will add the shasum variation to my answer soon.

[Admin]

great, thank you very much! Just a note: I was trying the "find TIFF -type f -name '*.tif' -exec md5 {} ';' >md5.txt" on linux and I'm getting "find: 'md5': No such file or directory" Do you know what's causing it? On OSX it works.

[Kusalananda]

@EmilyChoi Yes, md5 is not available on Linux. You may try md5sum there. Use md5sum --tag (I think) to get the same format output as on macOS. There's also sha1sum that generates the same kind of output as shasum on macOS. These Linux commands come from the GNU coreutils package and are often installed by default.

[Admin]

sorry, one more question raised up as I'm generating and checking md5 checksums: How can I write the output of the check into a textfile in linux? $ md5sum -c sums.txt > sums_checked_results.txt doesn't work

[Kusalananda]

@EmilyChoi Huh? That should work just fine. In what specific way does it not work?

[Admin]

yes, I was too impatient. After a few minutes the filesize of sums_checked_results.txt increased from 0 bytes. Thanks for helping!