File corruption after copying files in Windows 7 64 bit using two methods

Question

I have 5000 pictures and other files in a directory taking up 35 GB. I want to duplicate this directory.

Method 1: I do a simple copy and paste of the directory in explorer. I have the habit of checking the checksums after copying important files. In this case I noticed that around 2000 files failed the MD5 test. At a closer inspection of a randomly chosen JPEG with different checksums it turns out that some XMP metadata had changed. In particular, the tag

<MicrosoftPhoto:DateAcquired>

had changed the date from 2009 to today (possibly around the time I was copying the files). I have no idea what triggered this XMP data to be changed and exactly when it was changed and why for these particular files, but at least it seems to explain the checksum discrepancy.

Method 2: As I want the exact files to be duplicated, I tried the program FreeFileSync to mirror the directory, hoping no XMP metadata would mysteriously change. A checksum test in addition to a thorough file comparison test in FreeFileSync lead to two similar but yet different results: 31 files fail the checksum test, 23 files fail the file comparison test. The smaller set is not entirely contained in the bigger set, although many files occur in both. What is alarming here is that not only JPEGs are flagged as altered but also som AVIs, MPGs and a large 7-zip file. Closer inspection of a JPEG indicates that it is indeed corrupt: the bottom half of the picture is simply plain gray. Due to the size of the 7-zip file, I have not been able to pin down the discrepancy.

Note, in both methods, every file has its correct file size after being copied.

Question: Any thoughts on what is possibly going on here? I have never had this problem before, and I am now terrified that files get corrupted after simple actions like copy/paste and file sync. Even if I manage to successfully copy the files somehow, I would still like an explanation to this.

Solution 1

As well as checking your hard drive as ewanm89 suggested, you should check your RAM. Go to Control Panel, type in "memory" in the search box, and select "Diagnose your computer's memory problems".

You can also try other tools such as MemTest86.

Also check the System log in Event Viewer and look for errors.

Copying files really shouldn't change them.

Solution 2

After reading lots of articles on the web, I have a couple of knowledge.

1. Data corruption is usually due to bad RAM, followed by bad hard disks, failing PSU or bad motherboards. 98% of the time CPU is not the reason. That's why CPU is usually an expensive hardware.

2. ECC Ram is an advantage to avoid soft errors caused by RAM, but a non-ECC RAM is good enough for all purposes. Soft errors don't happen often.

3. A faulty PSU would usually lead to damage to motherboard and RAM. As a result, a faulty RAM or HDD is usually due to a faulty PSU, which should be replaced immediately. A bad or failing PSU would have a swelling capacitor and unpleasant burning smell.

4. A failing hard disk would generate an uncomfortable noise that would last for some time, and you would know there must be something wrong with it.

if anything wrong, please point out.

Solution 3

It's most probably a RAM problem. I had similar problems - moving files changed their checksums (very rarely, mostly in very big files at or above 1GB size and not always) and very often: archives were corrupt when created, mostly 7zip archives. So it seems like creating 7z archives with almost maximum RAM usage is a pretty good indicator of RAM problems. I ran Memtest86 and found a damaged bar, swapped it and had no problems since.

Comments

[Peter]

I am also seeing the <MicrosoftPhoto:DateAcquired> change in the file contents. Note that this is not file corruption, the files are still readable, but they are being modified while being accessed. For me, this is causing big problems with file backup (since every single JPEG file is now marked as changed and has to be backed up again). The problems in your second method do look like corruption, but that may also be caused by the application used.

[DustByte]

For the record: the disk is only a couple of months old (which doesn't mean it's not faulty). What are "the right options" for chkdsk? Could chkdsk delete/corrupt data on the disk?

[DustByte]

I just performed copy/paste from one physical drive to another, and it seems like it works. My checksum program indicated 19 errors but a couple of random checks indicate that the files are indeed the same! I need to investigate this further. I am still baffled why the synchronisation failed. What initially lead to me check checksums is that the backup program Areca refuses to back up my files as it fails on some hashs during checking - a problem I still need to resolve. The backup in this case was onto a third hard drive.

[J Slick]

@DustByte, go to Computer Management / Disk Management / Right-click on the disk / Tools tab / Check.

[DustByte]

Coincidentally, I had RAM problems when I bought the computer (brand new) a few months ago. I had the RAM replaced and used MemText86. My RAM is now fine. I will check the System log, and double check memory. Thanks.

[ewanm89]

On Unix systems there is a tool called rsync that can copy just changes to files there are ways to run it in windows however it is a command line tool with a lot of options that need to be got right. You might find it useful.

[DustByte]

Computer was shipped brand new only a few months ago. No BIOS settings have been changed. This set of four RAM modules were fine back in August. I run extensive memtests back then. Computer was initially shipped with faulty RAM, which I had replaced. Now I wonder what made the RAM go faulty all of a sudden? I can see there are manufacturing faults on brand new RAM, but why did this happen after a few months? Any thoughts?

[J Slick]

@DustByte, I don't know why RAM goes bad. It's surprising that this second lot would go bad, unless they were from a same bad batch. Or maybe that brand is incompatible with the motherboard. You could try reseating the RAM modules. That can sometimes fix things. Ground yourself first so you don't zap anything. Memory tests aren't completely definitive if they don't find problems. You should run them for some hours. There's still the possibility that the motherboard or perhaps the power supply are faulty.

[DustByte]

Thanks. Fortunately I will have new RAM shipped to me. They guy talked about issues with a whole batch from last summer, so hopefully I had RAM from that one, and the new RAM will be fine. From now on I will try to remember to routinely test my computer for hardware faults.

[tvdo]

> Earths magnetic field is constantly trying to realign the little magnets No, the planet's magnetic field is nowhere near strong enough to affect a HDD. And the kind of radiation you'd need to corrupt a HDD is likely to kill the entire thing.

[tvdo]

> Also this is the kind of errors raid helps avoid. No, this one of the reasons people say "RAID is not backup". RAID is only useful in the event of a full failure (read error, etc.). Subtle corruption, especially that caused by something other than drive hardware (in this case, RAM errors, but software faults can do the same thing), will corrupt data on the RAID array.