fsck won't fsck (unable to set superblock flags)

Question

Following an unclean shutdown on an SD card based device, I took the SD card out to fsck the root filesystem. This led to variations on the following:

e2fsck 1.43.1 (08-Jun-2016)
/dev/sdc2: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway<y>? no
Clear journal<y>? no
e2fsck: unable to set superblock flags on /dev/sdc2

Here I've answered "no" both times but there is no sequence of yes/no that does not immediately lead to the same outcome.

The filesystem can be mounted and on casual inspection appears okay; it also works fine in the device, and that's the root filesystem (actually it turned out to be not quite fine, see comments; tldr some irretrievably corrupted directories).

I dd'd the partition (8 GB) to a file, and tried fsck on that. Interestingly:

e2fsck 1.43.1 (08-Jun-2016)
plush.rootfs: recovering journal
Clearing orphaned inode 18290 (uid=0, gid=0, mode=0100644, size=34096)
Clearing orphaned inode 18270 (uid=0, gid=0, mode=0100644, size=38916)
Clearing orphaned inode 18250 (uid=0, gid=0, mode=0100644, size=1128076)
Clearing orphaned inode 11411 (uid=0, gid=0, mode=0100644, size=293108)
Setting free inodes count to 406127 (was 408580)
Setting free blocks count to 1305622 (was 1347486)
plush.rootfs: clean, 60209/466336 files, 604906/1910528 blocks (check after next mount)

A subsequent fsck passed clean, the image can be mounted, and fsck -f after that passes as well.

But the filesystem on the card from which the raw block copy image was created still has the same problem -- except that the systemd-fsck which takes place during boot logs the filesystem as "clean". Subsequently though, a proper shutdown, taking the card out, and trying fsck again from another box presents the same error.

Whenever the original is mounted on another machine, syslog notes:

kernel: EXT4-fs (sdc2): 4 orphan inodes deleted
kernel: EXT4-fs (sdc2): recovery complete

Since I have it all backed up, I'm open to trying anything here. I could simply forget about this and reburn the partition from the apparently fixed image, but that does not seem like a very satisfactory solution, since it means assuming fsck cryptically failed at solving a minor looking problem.

I suspect this is going to turn into a "request for official documentation" question regarding stuff like needs recovery_flag (or just plain "What does this mean?" question), so any suggestions along those lines are appreciated.

Solution 1

I just ran into this same problem. After debugging the issue with the e2fsck maintainer, we realised that the SD card was broken. It was accepting writes without error, but it wasn't actually writing the data to the card. The SD card was effectively read only.

It seems the card had gone into some sort of failsafe mode, where the data could still be read, but nothing written.

The e2fsck message unable to set superblock flags means it tried to write to the superblock to mark the journal as processed, which happened without error, but when it went to read the superblock back again it still indicated that the journal needed to be replayed. In other words, the changes written to the superblock were not saved on the storage medium.

The card I am using that has this problem is a Samsung Evo 16GB microSD, which I mention just in case it's a common problem with these cards.

I was able to test this by using dd to write 4096 bytes from /dev/zero onto the card at block 0, then I read back from the card and instead of getting all zeroes as I should, I still got the original unchanged ext4 superblock.

I'm now in the process of moving the data onto a new card and then seeing if I can get a replacement from Samsung, who appear to offer a 10 year warranty on SD cards.

UPDATE: Samsung replaced the 16GB card with a 32GB one in the same Evo series, so guess I can't complain too much!

Solution 2

I know this is an old thread, but, I figured I'd offer some insight.

This seems to be the way sd cards die a natural death. The number of read/write cycles sd cards can endure is substantially lower than most other mediums considered 'read/write'. When that has been exhausted, the card will go into read only mode, but won't inform you of such. Lots of things will think they're writing to the card thanks to OS caching, etc., but nothing ever sticks.

A great way to kill an sd card is to mount it as a swap partition or something that's very read/write intensive. You'd be surprised how quickly you can kill a card that way. I've found that running knoppix off of an sd card or usb thumb drive will only last a month or two, depending on the quality of the card and the intensity of knoppix usage. (I've since switched to running knoppix off of a usb SSD drive which has lasted a couple of years now).

Solution 3

Had the same issue but the problem was the physical read-only mechanism on my SD card adapter was enabled.

Solution 4

Had the same problem, unable to mount, unable to read superblock, unable to set superblock flags, unable to use superblock backup.

The problem I had was write lock switch on the card adapter ;)

View more solutions

Comments

[Admin]

@MarkPlotnick No, and it is writable. The last thing in the log from before the problem was systemd restarting (the device is headless and became unresponsive following a long apt upgrade). After that it logs a normal boot -- and the systemd-fsck says "clean" (I'll edit that in), but trying fsck outside of that context still fails.

[Admin]

Your fsck on the copy cleared 4 inodes, but fixed the free inode count by reducing it by 2453 inodes! That's enormous. Check that the device is getting enough power.

[Admin]

Doesn't it really suck when the very tool that's supposed to fix your problem, doesn't work because of the nature of the problem? Conclusion: The tool is bad and should be fixed.

[Admin]

Comment posted as (now probably deleted) answer from @TomásMayr : "Just came across this answer while trying to fix my Samsung Evo 16gb microSD, so to complement Malvineous, it might have something to do with this specific model."

[goldilocks]

"where the data could still be read, but nothing written" -> The fs was writable.

[Malvineous]

@goldilocks: Sounds like your fs superblock may not have been writable though. Also, my fs appeared writable thanks to caching, it was only after unmounting and remounting I noticed any changes were lost.

[goldilocks]

It was not an illusion due to caching.

[MoonCactus]

amazing: I have the same issue, with the same card ! looks like it is "read only" from now on, that is weird for sure

[MoonCactus]

@Malvineous thanks, this is good to know! I bought from Samsung specifically for the hope I was reducing such a risk, and it ends up the other way. It gets statistically significant enough for me to try something else... but what? May be it was only a batch of these 16GB cards... There might be no extrapolation to be made from one product to another even in the same brand!

[Lyokolux]

If you can, please provide a source for it. It would be great.