Read-only root file system - ext3 error?

Question

I have a problem with ext3 filesystem. It's not writable and I can't remount it.

# echo 1 > /file
-bash: /file: Read-only file system 
# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
# mount -o remount,rw /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only

Can I fix it without server restart?

Solution 1

It's probably dropped into read-only mode at the LVM level (lvs will lack a 'w' in the second caracter of the Attr column), so you might be able to flick it back with an lvchange --ignorelockingfailure -p w VolGroup00/LogVol00. Alternately, the VG may have dropped write access (typically caused by a hardware failure), in this case indicated by a lack of w in the first character of the Attr column in the output of vgs. That can't be changed at runtime, by the look of it.

If LVM has dropped write perms on either the LV or VG, it's probably for a very, very good reason, and unless you care not for your data, I'd be looking into the root cause rather than trying to bludgeon LVM into allowing writes again.

Solution 2

If this happened since the machine booted, check dmesg or other system logs for reports of anything terrifying going wrong with the hardware (e.g IO errors reported when accessing the underlying disks) such errors can trigger the kernel to put things into read-only mode until someone (you!) comes along to make a more informed decision.

Solution 3

There is probably filesystem error. Run fsck after umount. This wont require reboot. Try to fsck without using -a or -p.

Comments

[wazoox]

Anything in the logs about LVM problems?

[Govindarajulu]

Can you post output of cat /proc/mounts?

[Maciej Nowakowski]

Attr for this volume: -wi-ao, lvchange command returns "Locking type 1 initialisation failed."

[Govindarajulu]

Yes, it will require a reboot: it is his root filesystem :P

[Saurabh Barjatiya]

My bad. Should have read question properly. :'(

[womble]

Answer extended.