How can I calculate an MD5 checksum of a directory?

linux directory md5sum

158,428

Solution 1

find /path/to/dir/ -type f -name "*.py" -exec md5sum {} + | awk '{print $1}' | sort | md5sum

The find command lists all the files that end in .py. The MD5 hash value is computed for each .py file. AWK is used to pick off the MD5 hash values (ignoring the filenames, which may not be unique). The MD5 hash values are sorted. The MD5 hash value of this sorted list is then returned.

I've tested this by copying a test directory:

rsync -a ~/pybin/ ~/pybin2/

I renamed some of the files in ~/pybin2.

The find...md5sum command returns the same output for both directories.

2bcf49a4d19ef9abd284311108d626f1  -

To take into account the file layout (paths), so the checksum changes if a file is renamed or moved, the command can be simplified:

find /path/to/dir/ -type f -name "*.py" -exec md5sum {} + | md5sum

On macOS with md5:

find /path/to/dir/ -type f -name "*.py" -exec md5 {} + | md5

Solution 2

Create a tar archive file on the fly and pipe that to md5sum:

tar c dir | md5sum

This produces a single MD5 hash value that should be unique to your file and sub-directory setup. No files are created on disk.

Solution 3

ire_and_curses's suggestion of using tar c <dir> has some issues:

tar processes directory entries in the order which they are stored in the filesystem, and there is no way to change this order. This effectively can yield completely different results if you have the "same" directory on different places, and I know no way to fix this (tar cannot "sort" its input files in a particular order).
I usually care about whether groupid and ownerid numbers are the same, not necessarily whether the string representation of group/owner are the same. This is in line with what for example rsync -a --delete does: it synchronizes virtually everything (minus xattrs and acls), but it will sync owner and group based on their ID, not on string representation. So if you synced to a different system that doesn't necessarily have the same users/groups, you should add the --numeric-owner flag to tar
tar will include the filename of the directory you're checking itself, just something to be aware of.

As long as there is no fix for the first problem (or unless you're sure it does not affect you), I would not use this approach.

The proposed find-based solutions are also no good because they only include files, not directories, which becomes an issue if you the checksumming should keep in mind empty directories.

Finally, most suggested solutions don't sort consistently, because the collation might be different across systems.

This is the solution I came up with:

dir=<mydir>; (find "$dir" -type f -exec md5sum {} +; find "$dir" -type d) | LC_ALL=C sort | md5sum

Notes about this solution:

The LC_ALL=C is to ensure reliable sorting order across systems
This doesn't differentiate between a directory "named\nwithanewline" and two directories "named" and "withanewline", but the chance of that occurring seems very unlikely. One usually fixes this with a -print0 flag for find, but since there's other stuff going on here, I can only see solutions that would make the command more complicated than it's worth.

PS: one of my systems uses a limited busybox find which does not support -exec nor -print0 flags, and also it appends '/' to denote directories, while findutils find doesn't seem to, so for this machine I need to run:

dir=<mydir>; (find "$dir" -type f | while read f; do md5sum "$f"; done; find "$dir" -type d | sed 's#/$##') | LC_ALL=C sort | md5sum

Luckily, I have no files/directories with newlines in their names, so this is not an issue on that system.

Solution 4

If you only care about files and not empty directories, this works nicely:

find /path -type f | sort -u | xargs cat | md5sum

Solution 5

A solution which worked best for me:

find "$path" -type f -print0 | sort -z | xargs -r0 md5sum | md5sum

Reason why it worked best for me:

handles file names containing spaces
Ignores filesystem meta-data
Detects if file has been renamed

Issues with other answers:

Filesystem meta-data is not ignored for:

tar c - "$path" | md5sum

Does not handle file names containing spaces nor detects if file has been renamed:

find /path -type f | sort -u | xargs cat | md5sum

View more solutions

158,428

Author by

victorz

Updated on November 07, 2021

Comments

victorz over 2 years

I need to calculate a summary MD5 checksum for all files of a particular type (*.py for example) placed under a directory and all sub-directories.

What is the best way to do that?

The proposed solutions are very nice, but this is not exactly what I need. I'm looking for a solution to get a single summary checksum which will uniquely identify the directory as a whole - including content of all its subdirectories.
- luvieere over 14 years
  
  Take a look at this and this for a more detailed explanation.
- Noldorin over 14 years
  
  Seems like a superuser question to me.
- Hosam Aly over 14 years
  
  Note that checksums don't uniquely identify anything.
- jmucchiello over 14 years
  
  Why would you have two directory trees that may or may not be "the same" that you want to uniquely identify? Does file create/modify/access time matter? Is version control what you really need?
- victorz over 14 years
  
  What is really matter in my case is similarity of the whole directory tree content which means AFAIK the following: 1) content of any file under the directory tree has not been changed 2) no new file was added to the directory tree 3) no file was deleted
- zx8754 over 6 years
  
  Related post: How do I get the MD5 sum of a directory's contents as one sum?
Dan Moulding over 14 years

Should the last token be \;?
Ramon over 14 years

For subdirs use something like cat **.py | md5sum
ire_and_curses over 12 years

+1: Very interesting! Are you saying that the order might differ between different filesystem types, or within the same filesystem?
Dieter_be over 12 years

both. it just depends on the order of the directory entries within each directory. AFAIK directory entries (in the filesystem) are just created in the order in which you "create files in the directory". A simple example: $ mkdir a; touch a/file-1; touch a/file-2 $ mkdir b; touch b/file-2; touch b/file-1 $ (cd a; tar -c . | md5sum) fb29e7af140aeea5a2647974f7cdec77 - $ (cd b; tar -c . | md5sum) a3a39358158a87059b9f111ccffa1023 -
Valentin Milea over 12 years

Note that the same checksum will be generated if a file gets renamed. So this doesn't truly fit a "checksum which will uniquely identify the directory as a whole" if you consider file layout part of the signature.
Hawken almost 12 years

@CharlesB with a single check-sum you never know which file is different. The question was about a single check-sum for a directory.
Todd Owen over 11 years

This may fit some use cases, but generally you would want the checksum to reflect only the content and not the dates at all. For example, if I touch a file to change its date (but not its contents) then I would expect the checksum to be unchanged.
Sid over 11 years

ls -alR dir | md5sum . This is even better no compression just a read. It is unique because the content contains the mod time and size of file ;)
dsummersl over 11 years

@PuttySidDahari Nice solution. Its not truly finding all differences since its not computing md5s on the files themselves...but very fast and definitely 'good enough' for me!
Motsel over 11 years

Doesn't the tar utility create a huge amount of cpu overhead because of the compression ?! I think the above accepted answer is more efficient...
ire_and_curses over 11 years

@Daps0l - there is no compression in my command. You need to add z for gzip, or j for bzip2. I've done neither.
Michael Zilbermann about 11 years

you could slightly change the command-line to prefix each file checksum with the name of the file (or even better, the relative path of the file from /path/to/dir/) so it is taken into account in the final checksum.
Michael Zilbermann about 11 years

Take care that doing this would integrate the timestamp of the files and other stuff in the checksum computation, not only the content of the files
unutbu about 11 years

@zim2001: Yes, it could be altered, but as I understood the problem (especially due to the OP's comment under the question), the OP wanted any two directories to be considered equal if the contents of the files were identical regardless of filename or even relative path.
Michael Zilbermann about 11 years

@unutbu : I know; i was reacting to the previous note, from Valentin Milea.
localhost about 11 years

I'm pretty sure that this script will fail if filenames contain spaces or quotes. I find this annoying with bash scripting, but what I do is change the IFS.
segfault almost 11 years

@ValentinMilea just remove the awk ... part if you consider layout part of signature.
Konstantine Rybnikov almost 11 years

It didn't work for me, I think mainly because I copied files to external HDD, so their metadata attrs changed, and tar packs it also. Maybe tar has some options to skip metadata.
silvernightstar over 10 years

Is there a syntax error in your answer? I had to enclose the -name pattern in single quotes in order to get it to work.
unutbu over 10 years

@silvernightstar: For me (on Ubuntu/bash) it works either way but you are right; I probably should put quotes around it.
Adrian Frühwirth over 10 years

@unutbu Without the quotes it expands *.py and thus breaks if any .py files are in the current directory that you are running the command from, that's why it needs to/should always be quoted.
unutbu over 10 years

@AdrianFrühwirth: Thanks for the explanation.
fletom about 10 years

This is cute, but it doesn't really work. There's no guarantee that taring the same set of files twice, or on two different computers, will yield the same exact result.
brak2718 about 10 years

The problem is that the other directory you're comparing to could be on another machine with another filesystem, and tar has no guarantee about the ordering of how it bundles files. So you can have all the files individually have the correct checksum, but the tar|md5 computation will differ.
user3388884 about 10 years

Sorry I know this is a year old, but why sort the initial md5sums?
unutbu about 10 years

@user3388884: Consider two different machines with the same directory contents, but say, different names. The files may be listed in a different order. So the md5sums will be generated in a different order. We want to consider the two directories as equivalent, so we must come up with a canonical order for the md5sums before we hash the md5sums.
user3388884 about 10 years

ok i see... but you're sorting the md5sums, not the files... but i guess it wouldn't matter actually
Krzysiek about 9 years

My version of this command is: find /path/to/dir/ -type f -exec md5sum {} + | sort | md5sum | cut -c1-32. This take into account individual hashes and their paths. Checksum won't change, unless file will be renamed, removed or modified. Just always use the same path.
Gabriel Fair over 7 years

What parameters would I use if I only wanted to calculate the md5 checksum of a directory?
MUY Belgium over 6 years

Best answer can be found : unix.stackexchange.com/questions/35832/…
Hasitha over 3 years

tar c <dir_name> | md5sum, this is the ideal solution.
motzmann over 3 years

I'd rather replace the while-stuff with a plain xargs so -P for parallel processing is possible. This also requires an additional sort step for the second column because parallel md5sum is with no repeatable order. find "$dir" -type f -print0 | xargs -P 6 -r0 md5sum | sort -k2
alper over 3 years

Creating tar may differ for the folder due to its timestamp each time its generated and it won't be unique
Peter Mortensen over 2 years

What is it supposed to do? Can you elaborate in your answer (without an elaboration, this is not much more than a link-only answer)? (But without "Edit:", "Update:", or similar - the question/answer should appear as if it was written today.)
Peter Mortensen over 2 years

Why is cat required? Will it work for files with spaces in their name?
Peter Mortensen over 2 years

OK, tesujimath seems to have left the building ("Last seen more than 2 years ago"). Perhaps somebody else can chime it?
Peter Mortensen over 2 years

What is it supposed to do? What is the principle of operation? Why does it works? Can you elaborate in your answer? (But without "Edit:", "Update:", or similar - the question/answer should appear as if it was written today.)
Peter Mortensen over 2 years

OK, doesntreallymatter seems to have left the building ("Last seen more than 7 years ago"). Perhaps somebody else can chime it?
Peter Mortensen over 2 years

How does that overcome the problems with a defined sort order?
Abid H. Mujtaba over 2 years

If you don't cat the files the input to md5sum will be the output of find which is a list of file names (and paths) not the content of those files.
Putnik over 2 years

Note: I thought about omitting sort -u but we need it because otherwise order of the files can be different therefore the checksum as well.
Michael Shigorin over 2 years

Peter, I can't as I haven't used it much myself but rather chosen for inclusion in ALT Rescue images back in the day when I was the guy maintaining those; a simple link like that has helped me on SO so more than once... thank you for the query, anyways (only seen it today).
Stack Underflow over 2 years

What's the -r0 option to xargs? I see a -r option for not running the command if the input contains no non-blanks, but what's with the 0?
Tiago Lopo over 2 years

In case the path contains spaces, see the -print0 for find. we also have -0 for xargs and -z for sort, basically it will replace spaces by null characters.
Stack Underflow over 2 years

Oh, of course! You are combing the two different options -r and -0. I was thinking a single -r0 option. Thanks.