How can I fsck a partition when the device reads as busy (but has been confirmed otherwise)?

Question

I'm attempting to run fsck -p /dev/sda5 to repair errors on an ext4 partition, however the command outputs

fsck from util-linux-ng 2.17.2
fsck.ext4: Device or resource busy while trying to open /dev/sda5
Filesystem mounted or opened exclusively by another program?

I have confirmed using /etc/mtab and lsof that nothing is using the partition and it's not mounted. I also used fuser -k /dev/sda5 to forcibly close anything using the file, and umount to attempt to unmount it to no avail.

How can I force fsck to at least check, and hopefully to repair, the partition despite the fact that it reads as busy? Assuming I'm confident enough that it's neither mounted nor in use, and that the possibility of data corruption isn't an issue.

All commands were executed as root from an Ubuntu 10.04 32-bit liveCD. The partition is the system (non-home) portion of an Ubuntu 10.04 32-bit installation.

Solution 1

There are things (usually in the kernel, like the NFS threads, swap files, bind mounts, etc.) that can keep a filesystem busy that won't show up in fuser.

If you try to fsck a filesystem that is mounted, it will get corrupted. You should find a live CD that doesn't automatically mount your filesystems, like Knoppix or Fedora.

Solution 2

You can remove the partition from fstab and reboot the machine

Solution 3

Try booting another liveCD - I recommend SystemRescueCD that is designed specifically for solving problems with linux installations. It is also smaller than typical OS liveCDs. And to be on the safe side, run fsck from the console (no X).

If the standard fsck -p does not work, you can also try e2fsck -y -f.

Comments

[Keeblebrox]

I've found a method of solving the bigger issue related to this question, but I'm still looking for an answer as to how to force fsck if it thinks the device is busy (or why it can't be done).

[Bram]

I'm encountering the same problem on a RHEL5.8 system. /tmp has been unmounted, /etc/mtab and /proc/mounts show no trace of the file system. I've even rebooted the machine, made sure I'm the only one logged on. lsof shows no file in use on /tmp. The same procedure works fine on 4 similar servers installed using the same kickstart profile. The difference between those servers and this one is that this one has SAN storage attached to it. Could the multipath daemon cause this?

[Bram]

It didn't help in my case but the Red Hat KB (access.redhat.com/knowledge/solutions/54184) recommends adding the nodmraid kernel option as a workaround for this problem.

[Keeblebrox]

Everything has been run from a live disk so far. I'm trying to fsck a local filesystem, so I'd hope that NFS wasn't keeping it busy -- however, I get what you're saying about the kernel keeping it busy in ways that won't necessarily show up. Since it still happens in a live disk environment, though, do you know a way to stop the kernel from accessing it long enough for me to run fsck?

[Marda]

Yikes. In use from a live disk? I can only think of two things: 1) Try a different live disk (Fedora instead of Ubuntu, for example). 2) Check /etc/fstab on the real system. Maybe there's a swap file or a bind mount that is getting turned on automatically? Not necessarily in that order, I might add...

[Marda]

OK, somehow I totally missed the last line of the question. I've updated my answer appropriately.

[Keeblebrox]

+1 because your mention of auto-mounted swap led to an unrelated breakthrough and your responses have helped solve the bigger issue. I do still want to find a way to force fsck when I know that the device is neither busy nor mounted, and when data corruption isn't an issue.

[psusi]

Maybe mdadm has detected raid signatures on the partition and bound to it? Check /proc/mdstat.

[Keeblebrox]

According to the man page, e2fsck -y -f forces a check on a clean partition. It doesn't mention anything about mounted partitions. Do you know whether it will force a check then, as well?

[rozcietrzewiacz]

Probably not, but you should definitely avoid fsck-ing mounted filesystems (see the paragraphs at the top of e2fsck manpage). Another thought: if /etc/mtab seems to indicate the partition is not mounted, but you still get the error, you should also check /proc/mounts, as the mtab file could be outdated for some strange reason.