Server is not able boot /bin/sh no such file or directory

Question

We have ldap server configuration in centos 5.8. Server is working fine yesterday as we today restarted the server its not able to boot. Gives following error :

/bin/bash: ro : no such file or directory kernel panic- not synching:attempt to kill init

We tried to recover using rescue mode still no luck.

chroot /mnt/sysimage

/bin/sh file is available in directory still it gives same error.

---

This is a capture of the console showing the issue.

This is the content of /boot/grub/menu.lst file:

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-308.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-308.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
        initrd /initrd-2.6.18-308.el5.img

Solution

I think that the root filesystem was corrupted by an unclean shutdown and important files such as /sbin/init, /bin/sh and /etc/rc.d/rc.sysinit were damaged somehow. I am not completely sure about it, though.

I suggest you to reinstall bash, SysVinit and initscripts packages using the image CD. Assuming that CDROM contents will be available through /dev/hdb device node in rescue mode, the reinstallation process can be performed by using the following steps:

# mkdir -p /mnt/cdrom
# mount /dev/hdb /mnt/cdrom
# rpm --root /mnt/sysimage -ivh --replacepkgs \
    /mnt/CentOS/{bash,SysVinit,initscripts}-*.rpm

Alternatively, by inspecting the file /sbin/init, /etc/inittab and files under the directory /etc/rc.d, you could figure out why /sbin/init is calling /bin/sh and, afterwards, calling a binary named ro. This does not look normal.

PS: /bin/sh should be a symbolic link pointing to /bin/bash and not a regular file.

---

EDIT: once there is a confirmation that /sbin/init executable does not exist in filesystem, I am sure that such absence is the cause of the problem. CentOS booting process is probably executing /bin/sh as fallback and passing ro root=/dev/VolGroup00/LogVol00 rhgb quiet as command line arguments, thus leading /bin/sh to look for a script file named ro. Because it does not exist, /bin/sh throws the error message ro: no such file or directory and aborts.

I believe that reinstallation of the SysVinit package (which provides the /sbin/init executable) will fix the booting issue.

Comments

[Anderson Medeiros Gomes]

It sounds like a problem either on kernel's command line or in the initial ramdisk. I suggest you to check the integrity of ramdisk files in /boot/initrd-*.img. Also, please could you capture an image showing console's output and the content of /boot/grub/menu.lst? While capturing the image, please ensure that quiet and rhgb command line arguments are absent.

[Rakesh]

@AndersonMedeirosGomes Image URL: imgur.com/fXr1dS1

[dortegaoh]

The problem is not that /bin/sh does not exist, the problem is that /bin/sh tries to execute a binary named ro which doesn't exist. There is an error in your grub cmd line. Show your grub.cfg or menu.lst which you can find in /boot/.

[Rakesh]

@GeraldSchneider grub.cfg and menu.lst is same. " default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.18-308.el5) root (hd0,0) kernel /vmlinuz-2.6.18-308.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet initrd /initrd-2.6.18-308.el5.img " File url : easyupload.io/89w063

[Anderson Medeiros Gomes]

@Rakesh Thank you. I see that the error message appears after the root partition is mounted and, possibly, after /sbin/init is executed. Therefore, the cause of the problem is not in grub or in the initial ramdisk. It looks like a filesystem issue.

[Rakesh]

/sbin/init file is not available. How to recover or reinstall init file.

[Anderson Medeiros Gomes]

@Rakesh This is a very relevant information. Please, see my modified answer.

[Rakesh]

Thank you so much... able to boot now.

[Anderson Medeiros Gomes]

@Rakesh I am happy to help. I would also appreciate if you could signal that my answer had solved your problem by accepting it.