Can't write at all to hard drive or change permissions of anything as root

Question

This just happened on my server a few days ago, it's an Ubuntu server and there is only one hard drive on it. I am logged in as root, and I cannot write to any place on the drive. I can't even use chmod, or chown anywhere in the entire drive.

When I try something like sudo chmod -R -v 755 * I get an error on every single directory saying failed to change mode of <directory>: Read-only filesystem

I am the only one who uses and has access to this server, I have no idea what I did to make this happen but I really need to fix this.

Solution

First of all when you see a read-only filesystem error, your first step should always be to run an fsck (filesystem check) on the affected drive. That's because Linux has mounted the filesystem as read-only because it detected an issue on boot, and if that error was something like corruption then remounting as read/write could damage your disk beyond recovery.

Second of all, as Eyoung100 mentioned, you should NEVER run a recursive command on the root of your filesystem. You're extremely lucky that in this case Linux didn't let you actually execute that command or your Droplet would probably be messed up such that a repair would be unreasonably difficult for you.

Third you don't actually need the rw portion of eyoung's command, as it's the default in these cases. You can get away with just sudo mount -o remount / which will attempt to remount your filesystem.

Typically the write-protected error isn't the problem with these, that is odd. But you should be able to get around it by running:

blockdev --setrw /dev/vda1

Then try the previous command:

mount -o remount /

Please let me know what you see when you try that. If it doesn't work, you'll probably need to request the recovery ISO and just migrate your files off the Droplet.

Comments

[eyoung100]

Is your file system local or remote?? It looks as if you need to remount the Filesystem as read/write. What is the result of sudo mount -o remount,rw / NEVER EVER use the command you posted, because if that does succeed, the only way to reset the permissions to the default is a full reinstall.

[chrispytoes]

@eyoung100 Thanks for the information, and using that command gives me mount: cannot remount block device /dev/vda1 read-write, is write-protected

[eyoung100]

Then you need to remove the write protection on the drive. I'll leave that up to you, as it's BIOS and drive dependent.

[chrispytoes]

@eyoung100 Also it's a remote server hosted on DigitalOcean.

[chrispytoes]

@eyoung100 Sorry how do I do that?

[eyoung100]

In that case contact digital ocean, and have them do it, and if I were you, I'd have them reset your VPS, and start fresh. That is a common occurrence for hackers who gained root access remotely...Never enable root access through SSH, always use sudo. If you need root access use sudo su or better yet sudo -i

[chrispytoes]

@eyoung100 Wow alright so I checked the graphs on my Digital Ocean account and I see a spike in CPU and bandwidth usage during a time when I was definitely not using it. I'm rather new to this, I was logging in with root over SSH. I'll contact Digital Ocean. Any specific tips so this doesn't happen again? Also before it's reset I need to grab some files I need from it so how can I do that without permissions? I can't even ftp into it anymore.

[eyoung100]

You can't... Whatever you have there, the hackers locked you out of. As such, it stays locked, unless you can have Digital Ocean login to your hosting VPS locally as root. In cases like this, it's better to leave it locked and start over so the damage doesn't spread. See Why is root login via SSH so bad that everyone advises to disable it?

[chrispytoes]

Wow okay, so doing fsck and rebooting did actually fix everything haha. But can you give me any insight as to what actually happened?

[eyoung100]

@chrispytoes I'm glad it wasn't an attack. Had this not been answered, I was going to do a writeup on how to disable remote root access via SSH. Instead, see How do I disable remote SSH login as root from a server? Put your user in the wheel group, and add said user to the Allow List as per the link, and from now on, use sudo -i

[Darian]

This particular issue (read-only filesystem) is very common with upgrades from 14.04 to 16.04. Did you recently run a dist-upgrade? I'd also like to note that running a dist-upgrade on a production server is never a good idea, you should instead migrate to a newer server if you must use the newer version.