Does Ubuntu damage USB drives?

Question

Whenever I unmount a USB drive with Nautilus and plug it to a Windows machine, I receive a warning that the drive needs to be repaired.

(the same message like in this screenshot)

For 10+ years working with windows I never had a corrupted or damaged USB drive but during the last two years three of my USB drives became inoperative, so I cannot prove it, but it's obvious that this is related to Ubuntu's (un)mount behavior.

A friend told me I can prevent such damage using udisks and sync, but I hope this is not the way to do it, mounting drives with shell commands in 2016.

Solution 1

No worries Ubuntu did not damage your USB drive. But we do not use poorly documented bit flags of a FAT32, FAT16, or NTFS filesystem. On Windows these flags indicate a possibly corrupted filesystem when we had not properly unmounted the drive or an I/O error had occured.

Those bits are located in a reserved entry of a FAT partition table. According to an internal Microsoft paper of 2004 the purpose of these bits are:

• ClnShutBitMask:
  If bit is 1, the volume is “clean”. The volume can be mounted for access. If bit is 0, the volume is “dirty” indicating that a FAT file system driver was unable to dismount the volume properly (during a prior mount operation). The volume contents should be scanned for any damage to file system metadata.
• HrdErrBitMask:
  If this bit is 1, no disk read/write errors were encountered. If this bit is 0, the file system driver implementation encountered a disk I/O error on the volume the last time it was mounted, which is an indicator that some sectors may have gone bad. The volume contents should be scanned with a disk repair utility that does surface analysis on it looking for new bad sectors.

There was some discussion with the kernel file system developers some years ago on how to overcome this but I was unable to follow up the results. Apparently it did not make it into recent kernels.

As only a bit flag is involved but generally our data should be in perfect shape we can ignore the warning Windows issues on those drives we had previously used in Ubuntu.

Solution 2

This is mostly an issue with Windows. It thinks it is the only OS in the world and acts out if it detects something it does not understand.

Just because Windows says you must repair the drive does not make it true.

Any of my drives used with Ubuntu get this message from Windows, I just say no and they work fine with Windows.

In short there is nothing wrong with the drive, it is just there is something on there that Windows does not understand and its response is destroy it.

Don't repair the drive, it will format it without telling you and you will lose all the data on the drive.

Solution 3

As you suggested in a comment, this is possibly related to Ubuntu not completing the write process before you remove the flash drive. Ubuntu writes the files to RAM during the copying process, and writes these files from the buffer to the USB drive in the background after the copying dialog is closed. There's another answer that suggests this is especially a problem for machines with large amounts of memory, but it's likely that you're removing the drive before the background write is complete. When you eject a drive, this forces the buffer to be written to disk, but it is difficult to tell when the write is actually completed. There's a pop-up in newer versions (15.10 for certain, possibly 15.04) intended to remedy this -- a purple alert appears after some (generally short) time that says something along the lines of "[Drive] can now be removed" once the write is finished.

This is probably the reason your friend suggested sync. Running sync in terminal forces all buffered data to be written to disk, and should return only once the writes are completed (See this answer). See this answer for a utility that will help you determine if this is actually the problem.

Solution 4

I've seen lots of usb drives, especially flash (SD, etc) cards in USB adapters, with FAT filesystems (virtually every USB drive comes pre-formatted with FAT32) getting corrupted while using Ubuntu & Ubuntu-derived distros. It was almost a regular occurrence every few months of regular daily use.

If it were a hardware problem, then formatting the partitions with ext3 (or 4) shouldn't help, but ext3/4 makes the filesystems virtually bulletproof. (ext2 also was corrupted over the few months I tried it; was supposed to have less writes so a longer life for flash drives, mainly from no journal).

So, Ubuntu should not physically damage USB drives themselves, but I still don't trust it 100% with FAT filesystems.

I think filesystem corruption can be best avoided by:

• Never just unplug a USB drive until after it's been unmounted/umount/ejected. Even if the drive's not currently writing anything (if it has a light it's not flashing) and even if you've done a sync the filesystem may still
• Wait at least a few seconds after umount/eject before unplugging it. It seems that even after umount the drive's activity light may sometimes keep flashing for a little while. Other users like this guy say it could go on for up to a minute.
• Don't only rely on only sync either, like this guy who got file corruption.

Related (generally) links:

• Answer to "Should I unmount a USB drive before unplugging it?"
• Is execution of sync(8) still required before shutting down linux?
• How to remove a USB drive without worrying if its been unmounted?

Solution 5

This has little to do with any operating system. This is a file system error, which in most cases indicates that the drive was not unmounted properly (user error). It is not an absolute indication of damage, it merely indicates a possibility of a damage. If present, the damage is usually limited to the last written file(s). Windows "Repair" command in this case scans the drive for sectors marked as "occupied", but not assigned to any file and then assigns this scraps and pieces to new files.

This error can be also created on Windows, if you just yank out the drive during write without unmounting it first (in Windows "Eject" performs the unmount).

Seeing this message means that the drive was not unmounted properly.

You can verify on your own if the file system was really damaged or is it just false positive: if after running "Repair" a new folder named like "found000" appears with some files in it, it means there really was an unfinished write that got interrupted. Some of your data were lost, and that is a concern.

View more solutions

Comments

[arielf]

I would try two tests to figure out the root-cause: 1) Can you prevent the error from appearing if you call sync in a shell before unmounting in Ubuntu? (unlikely) 2) Try writing only one file at a time, unmount, then check if this file can be read on windows and its contents are up-to-date, despite seeing this warning. 2) means the issue is most likely with Windows being fussy saying "we found errors" about some non real issue.

[user541686]

Don't believe people when they tell you Linux won't corrupt your drives. It can. On Ubuntu 15, mere file copy followed by reboot consistently gave me errors in chkdsk like this: Stage 2: Examining file name linkage ... Found corrupt basic file structure for "<0x32,0x1e63>" ... queued for offline repair. Found an unneeded link ($FILE_NAME: ???) in index "$I30" of directory "\ <0x5,0x5>" ... queued for offline repair. Found missing Index entry for file "<0x32,0x1e63>" from index "\??\D:\found.000" of directory "$I30" ... queued for offline repair.

[user541686]

^...not to mention, this wasn't even on a flash drive, it was on my main SSD. A brand-new SSD. And it happened every single time that I didn't add a ton of extra I/O after the file copy. In other words, Linux's implementation of NTFS is broken, as much as its fans will want to deny it and avoid believing.

[Braiam]

Is this a NTFS filesystem? FAT32? Can you reproduce it reliably?

[Tracy LF]

I've noticed that if you wait a few (say, 15) seconds after you eject/unmount the USB & then remove it, you don't have this trouble. It seems that even though it shows immediately after unmount that it's "ejected", it actually is still busy?

[ulidtko]

What's so bad with mounting drives with shell commands ? ¯＼(o_o)/¯

[MatthewRock]

I have the same behavior the other way - I must 'repair' USBs under Linux, because Windows gets them broken.

[TMN]

Are you unmounting the drive, or just pulling it out? If you're properly unmounting it, then all pending writes should be performed and the disk buffers flushed. If you're just pulling the drive out, then I'm not surprised you've got file system corruption.

[Ken Mollerup]

No no no ! this is a windows scam project started with windows 7 servicepack 2 put in production with windows 8: It works like this.. Anything accessing a forign ( not exclusively Windows Owned drive - may it be a USB or iscsi Drive (HD, stick memory card) that have been accessed by something not OWNED BY Microsoft Will trigger all KINDS of SCARY WARNINGS and ultimately tell you to format the drive ( before reading / Copying of course)!! Next time try it on a Unix, Machintosh or other Linux distro ( same result - No Complaints What so ever!!) Beware of Microsuft Agressive Marketing. !!! :-(((

[Frédéric]

@Mehrdad, linux NTFS implementations have very long been experimental when it was about writing. The main trouble with NTFS is its closed spec aspect, meaning Microsoft does not give much information on how to implement NTFS. This is not Linux fault but Microsoft choice. About current NTFS implementations in Linux, there is many different ones. One or many of them are maybe broken about writing, or just obsolete in your current system. Look here for a list under Debian. I guess there is equivalent choices elsewhere for Ubuntu.

[user541686]

@KenMollerup: Tell us why anyone should believe what you're saying when you have provided zero links or technical details.

[user541686]

@Frédéric: It's Linux fault and Microsoft choice. You can't corrupt user data and then blame it on someone else making their software closed-source. Furthermore, I believe Microsoft does share details of NTFS with its "partners" (those who make partition managers etc.), they just don't do it for free, but for a commercial company like Canonical, it doesn't seem out of the question to try that route. Finally, how do you even know the incompleteness of the current reverse-engineered spec of NTFS is the source of the problem? To me it seems like a completely unrelated bug.

[Frédéric]

Continued in chat.

[Ken Mollerup]

@Mehrdad I Have 2 Usb sticks that after Windows tried to use them for backup, while I was trying to put my Mail back together, show 0 bytes awailable out of 0! Nothing left out of 16GB Kingston DataTraveler, and a NoBrand 8GB. Nothing can make them come back to life!

[JinSnow]

related: askubuntu.com/questions/955565/…

[jan6352781]

1. Repair does not format, repair saves allocated data to a folder. For me it seems as Ubuntu does not finish write process properly.

[jan6352781]

2. Most USB drives are FAT32 formatted, which was developed by Microsoft, so why should Windows be the only system to "not understand" somethin??

[jan6352781]

3. The last years I skipped the repair almost every day ending up with damaged files and USB drives.

[Sergey Kanaev]

@jan6352781 (2). Nertheless, when you put a hlt x86 instruction as the first byte of the USB-drive Ubuntu, and any other GNU/Linux does so too, will not consider this a fault while Windows, at least XP does, says that the drive is currpted and needs a format. FYI the first 512 bytes of a drive consist of 442 lower bytes of bootloader and 72 upper bytes of partition table. Thus, chainging the first byte of USB-stick only changes boot loader to a custom one (not provided with OS or third-party software).

[Jon Bentley]

I get the popup in 14.04 (Linux Mint KDE edition) whenever I eject a USB drive, and also my 15.10 Kubuntu machines. Not sure about 15.04, but I would assume so.

[Thomas Ward]

I've never had these errors in these drives when I actually safely unmount the drives - are you sure support for these bits is missing?

[Izkata]

The GUI really doesn't make it obvious? I've been using eject in the terminal for a long time, and there's a noticeable delay before it returns if I've just copied a large file.

[alex_d]

@Izkata On vanilla Ubuntu 14.04, when you click the eject symbol in Nautilus it disappears immediately. Unlike internal drives, though, USB drives disappear completely from the sidebar when completely unmounted (i.e. write is finished), so there's some amount of time during which the USB drive is being written but appears to be ejected.

[wizzwizz4]

The OP is worded badly, but it means "file system", not "flash ROM".

[Takkat]

@ThomasW. These bits are quite obscure but I occasionally have these errors with my Ubuntu-formatted USB drives on my workplace Windows 7. So far I haven't seen any helpful pattern. Only thing I can say is that I always properly unmount/eject the drives. Testing is not easy because at home no Windows, at work no Ubuntu.

[Fabby]

Yeah, Windows 10 formatted my EXT4 Lubuntu partition because it cabe before the FAT partition. When the FAT partition comes first, there is no problem...

[user541686]

-1 because it's unfounded and wrong. I have seen Ubuntu 15 corrupt my NTFS volume (yes, I know you won't believe me, but I've seen it with my own eyes and reproduced at least 3-4 times in a row before believing it myself), and it has happened even despite me explicitly sync'ing all the buffers. I concur with @jan6352781 that I also suspected it was due to not finishing the write process, and I still do. In fact, if you wait long enough after the write before rebooting Ubuntu, it works fine. We have no reason to believe Windows is at fault here, and every reason to believe Ubuntu is.

[Mark Kirby]

@Mehrdad I don't see how Ubuntu is at fault because you did not let the writing finish on the drive "unfounded and wrong" You are wrong about this because it is right that that warning appears in Windows due to a lack of compatibility between Windows and other OS's, I see your point but you choice of words (particularly unfounded as you can see many agree) is very poor.

[Xen2050]

It appears that the OP may mean actual USB drive damage, they say " I never had a corrupted or damaged USB drive but during the last two years three of my USB drives became inoperative" If they meant just corrupted they probably would have said corrupted, instead of "inoperative" @wizzwizz4

[wizzwizz4]

@Xen2050 Yet all of the other answers, and comments threads including jan6352781 imply, or state, the file system is the issue, and the "broken device" is an extrapolation of them not working properly.

[Xen2050]

@wizzwizz4 Well, in Mark Kirby's answer, the OP comments "3. The last years I skipped the repair almost every day ending up with damaged files and USB drives. " I'm guessing it's just from old drives that would have failed, windows or no, but they OP still said it

[wizzwizz4]

@Xen2050 Or maybe the OP was referring to the old message "The device is damaged. Do you want Windows to fix it?" (paraphrased) which sometimes comes up when a different dll handles the issue. (I've had it appear myself, although I'm not sure the last sentence is right.)

[wizzwizz4]

I'm not saying this answer is wrong, and I'm not downvoting it (I've upvoted actually), but I was just providing insight into the interpretation of the question that everybody else, including the question poster, used.

[Xen2050]

@wizzwizz4 That may be true. Unfortunately the question (like most questions ;-) isn't crystal clear. (I upvoted this one too, thought we were the only ones, but it's +4 and -2 right now)

[web.learner]

Comments are not for extended discussion; this conversation has been moved to chat.

[EugeneK]

Repairing the drive does not format it. It runs chkdsk. It no more formats the disk than fsck formats a disk.

[Millie Smith]

This entire answer is just a Windows rant with zero evidence (except anecdotal).

[MatthewRock]

The headline is false.

[Takkat]

@MatthewRock: better?

[MatthewRock]

Yes, it is better.

[xji]

This seems to be the more helpful answer.

[Ken Mollerup]

Oh Yes Microsoft Windows have quite a few: F.eks Wait til you start copying a large file to a (Quite Big Enough) Usb stick - then autostart the Backup Anything on the Whole HD! It will shurely destroy any Usb- device with less space than required by this aforementioned operation!!!

[Wilf]

Whenever I had problems I just check and fix the drive with GParted - often it says 'Removing dirty bit' which I suppose refers to the above. These problems I usually had with silly windows machines refusing to unmount, or newer USB drives where you have to use 'Safely remove drive' instead.