System went read-only suddenly

Question

My web server (Linux, Debian Wheezy) has suddenly gone read-only, MySQL have crashed (but Apache didn't) - giving errors when pressing tab to complete file name as:

bash: cannot create temp file for here-document: Read-only file system

Also, when I restart Apache it won't turn on anymore.

What I tried to do : I've replaced the hard disk , moved web files again ; error repeated after 1 week.

Logs shows nothing, one notice that I've seen many failed attempts to access the server through ssh.

df output:

Filesystem                                             1K-blocks    Used Available Use% Mounted on
rootfs                                                 249773956 2328024 234758164   1% /
udev                                                       10240       0     10240   0% /dev
tmpfs                                                     406336     200    406136   1% /run
/dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c 249773956 2328024 234758164   1% /
tmpfs                                                       5120       0      5120   0% /run/lock
tmpfs                                                    2489760       0   2489760   0% /run/shm
/dev/sdb1                                              249773956 2303784 234782404   1% /mnt/sdb1

mount -n output:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=506431,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=406336k,mode=755)
/dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c on / type ext4 (ro,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=2489760k)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sdb1 on /mnt/sdb1 type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)

Solution

Your root file system is mounted read-only. This likely happened on a reboot. There are a few options:

• Configure the system to fix errors during reboot. On Ubuntu this is controlled by the FSCKFIX option in the file /etc/default/rcS.
• Reboot in recovery mode and run fsck -f /dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c and reboot again.
• Boot from a recovery disk and run fsck as shown above.

All options will require your system to be down for a bit. However, it appears it is effectively down now.

It may be possible to run fsck without rebooting and remount the system rw. I would run a test fsck without enabling fixing problems to see how broken the file system

Comments

[gogoud]

these many ssh attempts that you report, are they by you (or some other expected person)? If not, is it right that a third party can get ssh access to this machine? Any evidence that they have succeeded in logging in? If it isn't mindless (or worse mindful) vandalism by a third party, it seems there must be an error occurring on your / mount, then it remounts read-only. I would boot the system from CD/USB (e.g. systemrescuecd) and then do e2fsck -fp /dev/disk/by-uuid/e45e30eb-efa4-4cd9-aaf9-c6cbe46aa41c

[Ahmed Yusuf]

@gogoud No there are no successful attempts except for mine, and fsck won't be helpful since it's a new harddisk even

[Thomas Dickey]

Alternatively, it can happen without a reboot if something breaks one of the kernel modules supporting the file-system journalling. Rebooting is a workaround in that case.