How to recover from a chmod -R 000 /bin?

Question

And now I am unable to chmod it back.. or use any of my other system programs. Luckily this is on a VM I've been toying with, but is there any way to resolve this? The system is Ubuntu Server 12.10.

I have attempted to restart into recovery mode, unfortunately now I am unable to boot into the system at all due to permissions not granting some programs after init-bottom availability to run- the system just hangs. This is what I see:

Begin: Running /scripts/init-bottom ... done
[   37.062059] init: Failed to spawn friendly-recovery pre-start process: unable to execute: Permission denied
[   37.084744]  init: Failed to spawn friendly-recovery post-stop process: unable to execute: Permission denied
[   37.101333] init: plymouth main process (220) killed by ABRT signal

After this the computer hangs.

Solution 1

Boot another clean OS, mount the file system and fix permissions.

As your broken file system lives in a VM, you should have your host system available and working. Mount your broken file system there and fix it.

In case of QEMU/KVM you can for example mount the file system using nbd.

Solution 2

Even as root, you can't execute files that have no x permission bit set. What you can do though is call ld.so on it (provided they're dynamically linked executables):

$ echo /lib/*/ld*.so
/lib/i386-linux-gnu/ld-2.27.so /lib/x86_64-linux-gnu/ld-2.27.so

Use the one that matches the architecture of chmod executable. In my case the x86_64 one:

sudo /lib/x86_64-linux-gnu/ld-2.27.so /bin/chmod 755 /bin /bin/chmod

Or call something in /usr/bin or elsewhere to do the chmod like perl:

sudo perl -e 'chmod 0755, "/bin", "/bin/chmod"

Beware when restoring permissions that some files in /bin like mount or su are meant to have permissions other than 0755.

If you've rebooted, however, you might not be able to get to the point where you can run perl or ld.so though. You can fix things from the initramfs though (pass an incorrect root directory to get a recovery shell in the initramfs; see also the break=bottom or break=init kernel parameter on Debian, for the initramfs to give you a shell after the root file system was mounted (read-only though)). Or boot your VM from a live CD image, or fix by mounting the VM file system on the host as others suggested.

Fixing the initramfs way:

In grub, edit the boot entry and remove the root= parameter from the linux command:

setparams 'Ubuntu, with Linux 3.2.0-27-generic'                          
                                                                         
recordfail                                                               
gfxmode $linux_gfx_mode                                                  
insmod gzio                                                              
insmod ext2                                                              
set root='(hd1)'                                                         
search --no-floppy --fs-uuid --set=root dc02b07c-88ef-4804-afe0-4f02db2\ 
94561                                                                    
linux /boot/vmlinuz-3.2.0-27-generic                                     
initrd /boot/initrd.img-3.2.0-27-generic                                 
                                                                         

Ctrl-X to boot. Ubuntu's initramfs won't find the root file system so start a recovery sh. Then mount the root filesystem (in my case /dev/vdb, adapt to your machine) and fix things there:

Target filesystem doesn't have requested /sbin/init.
No init found. Try passing init= bootarg.


BusyBox v1.18.5 (Ubuntu 1:1.18.5-1ubuntu4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) mkdir /x
(initramfs) mount /dev/vdb /x
[   48.430071] EXT3-fs (vdb): error: couldn't mount because of unsupported optio
nal features (240)
[   48.477406] EXT4-fs (vdb): recovery complete
[   48.477747] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: (
null)
(initramfs) chmod -R 755 /x/bin
(initramfs) umount /x
(initramfs) reboot

Once booted, fix the permissions of the files that are not meant to have 755 permissions by comparing with another system.

Fixing by running python as init:

In grub, edit the boot entry, this time keep the root= parameter, change ro to rw and add a init=/usr/bin/python:

setparams 'Ubuntu, with Linux 3.2.0-27-generic'                          
                                                                         
recordfail                                                               
gfxmode $linux_gfx_mode                                                  
insmod gzio                                                              
insmod ext2                                                              
set root='(hd1)'                                                         
search --no-floppy --fs-uuid --set=root dc02b07c-88ef-4804-afe0-4f02db2\ 
94561                                                                    
linux /boot/vmlinuz-3.2.0-27-generic root=UUID=dc02b07c-88ef-4804-afe0-\
4f02db294561 rw init=/usr/bin/python
initrd /boot/initrd.img-3.2.0-27-generic                                 

Then, at the python prompt:

Begin: Running /scripts/init-bottom ... done.
Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.chmod('/bin/sh',0755)
>>> os.chmod('/bin/chmod',0755)
>>> os.execl('/bin/sh','sh')
sh: 0: can't access tty; job control turned off
# chmod -R 0755 /bin
# mount -o remount,ro /
[  100.704720] EXT4-fs (vdb): re-mounted. Opts: errors=remount-ro
# exec /sbin/init

Again, once booted, fix the permissions of the files that are not meant to have 755 permissions by comparing with another system.

Solution 3

Use python :)

$ python
>>> import os
>>> os.chmod('/bin', 0755)

That shouldn't need anything from /bin to do its job. Obviously, I haven't tried this out...

Comments

[jett]

I think this is probably correct as a way to fix it, however, I need to try to get the system mounted- right now I am just getting the presystem from the image file- as in initrd.img memtest & abi.

[Olivier Dulac]

+1, another great answer, Stephane. I'd add: in the already try to reboot case : boot on a live CD, mount rw the partition containing /bin, and chmod 755 /bin (and files inside if they were changed too). But afterward, check that all files are the right permission (depending on your linux distrib, you probably can check /bin against the original package)

[Stéphane Chazelas]

@jett, you mounted the /boot partition of that VM. Try and locate the root file system. If on LVM, run vgchange -ay after connecting the nbd to activate it.

[jett]

@StephaneChazelas I've got it. Thank you both so much- I love these kinds of mistakes, have learnt a ton!

[jett]

Hmm, I always figured that this and other scripting languages would just call the chmod program. Pretty good to know!

[Dennis Kaarsemaker]

No, chmod is a system call, called by the chmod program and also by the chmod function in python/perl/ruby etc. shells do call the chmod utility though.

[slm]

WOW. Your depth of knowledge is scary 8-).

[Stéphane Chazelas]

Except those shells that have chmod builtin. That's the kind of situation where shells like sash are useful. It's statically linked and has most recovery commands like chmod builtin (so doesn't rely on anything else). It would usually sit in /sbin though it wouldn't harm having extra copies on all the file systems, and can be used in combination with memlockd. zsh and ksh93 have a chmod builtin (though not enabled by default).

[Nadir Sampaoli]

@Dennis But how can you run it if you cannot boot into the system at all? As the OP says: «unfortunately now I am unable to boot into the system at all».

[Abhijeet Rastogi]

I can't use ld.so to execute. pb.abhijeetr.com/fRWf What's going wrong here?

[Stéphane Chazelas]

@shadyabhi, you probably have a multiarch system and are trying to use the 32bit ld.so on a 64bit executable. You must have another ld.so, maybe in some directory like /lib/x86_64-linux-gnu.

[Abhijeet Rastogi]

@StephaneChazelas Thanks. It was indeed the case. The correct path in Archlinux is /usr/lib/ld-2.17.so

[gokhan acar]

Glad it's fixed. I'm missing something. If it's a vm, then it's just one big file to the host system, isn't it? How do you mount anything inside it to do the repairs? (I understand how to do this on a non-vm system.)

[Chris Warrick]

I suggest doing a reboot instead of running /sbin/init manually. Stuff might go apeshit otherwise (PID 1 = /usr/bin/python).

[Stéphane Chazelas]

@Kwpolska, os.execl and exec are for executing, they don't fork a process, just replace the executable in the same process, so all is done in pid 1. Process 1 initially runs python, then sh, then init.

[Ken Bellows]

@NadirSampaoli This is definitely one of those things that you need to catch before you reboot. This could be successfully done after chmodding root and before shutting down the system.