Stuck at grub prompt after a default CentOS 6 installation

Question

I am having a very strange problem installing CentOS 6 that I've never encountered before. I've followed these same steps thousands of times before, but within the past couple weeks I've had two separate servers that would boot to a grub prompt after a fresh CentOS 6 install. Here are the steps of what I've done:

1. I download the CentOS 6 net-install ISO to a Windows file-share.

2. I use the IPMI web interface on my Supermicro server to mount the ISO as a USB virtual CD-ROM.

3. I then proceed to install CentOS 6 with the default partitioning and default boot loader (installed into the MBR). At this point, everything is normal.

4. I unmount the virtual CD-ROM and reboot the server to complete the installation. Instead of booting up CentOS, it goes to a GRUB version 0.97 prompt.

5. I use the grub command root (hd0,0) to attempt to select the correct disk and then boot into the operating system, but when I type that command I get an Error 21 that the disk does not exist. I also try running root (hd1,0) thinking that the system labelled the primary drive as /dev/sdb during the installation. No such luck -- I get a disk read error message.

6. At this point I reboot the server and use the install CD to get into rescue mode. I use rescue mode to inspect /boot/grub/device.map and /boot/grub/grub.conf -- everything is exactly as it should be -- (hd0) is mapped to /dev/sda and /boot/grub/grub.conf points to (hd0,0) for the boot partition. An fdisk -l command confirms that the boot partition is indeed setup on /dev/sda1

7. For good measure, I ran grub-install /dev/sda to re-install grub and rebooted the server. After I did that, I was no longer able to get to the grub prompt -- instead my server would just display an Error 21 error message indicating the selected disk does not exist.

8. I then proceeded to swap out hard drives and use a CentOS 6 net-install ISO hosted on a different mirror, repeated the steps above and got the same issue each time.

Screen capture of my grub.conf, device.map and fdisk -l from rescue shell: http://pbrd.co/1ijpSke

Screen capture failing to select (hd0,0) from grub prompt: http://pbrd.co/1ijr33d

I am able to install other operating systems on this server (I installed Ubuntu shortly thereafter without issue), but not CentOS 6.

I'm stumped where to go next. Any ideas?

Solution

I eventually resolved this problem by following the advice given by Michael above. Zeroing out the first few megabytes of the drive and then re-installing the operating system did the trick. I suppose there was some sort of MBR or partition table corruption at play.

If you are stuck at a grub screen after a fresh CentOS installation, try following these steps:

1. Insert the CentOS installation disc or mount the ISO.
2. Boot into rescue mode and enter into shell. Skip any steps to mount the existing file-system.
3. Run the command fdisk -l to determine the label of the drive you need to zero out (e.g., /dev/sda, /dev/sdb). If you have multiple drives, be very careful to pick the right one.
4. Run the command dd if=/dev/zero of=/dev/sdX bs=512 count=4000 where /dev/sdX is the drive in question (e.g., /dev/sda, /dev/sdb).
5. Exit out of the rescue shell and reboot.
6. Re-install CentOS 6 as per normal.

Comments

[Michael Martinez]

remove your device-map file and rerun grub-install

[Elliot B.]

I'm loading up the rescue shell to try that now, but could you explain how that will work? As I understand it, the contents of my device.map are fine.

[Michael Martinez]

grub-install will re-read the devices, so it's the easiest first thing to try. If that doesn't work, then the problem is either (a) bad network connection during some part of the install; or (b) bad hardware.

[Elliot B.]

I get the same result as before when I run grub-install /dev/sda without removing device.map. I rebooted, grub prompt is gone, but in its place is an Error 21 message -- selected disk does not exist. I've experienced this error on two completely different sets of hardware (identical server configuration though). I suppose a network issue could be to blame, but I've repeated the CentOS install so many times that I doubt that's the issue. Ubuntu net-install works fine.

[Michael Martinez]

zero out the first couple megs on each hard drive and try again. if it still does'nt work, then run tests on the RAM, hard drives and motherboard of the two servers in question.

[Michael Martinez]

do you know how to test hardware on supermicro servers? Use memtest for memory; check the SMART codes for the hard drives and use smartctl to run self-checks on the drives from within Linux; and use "stress" (people.seas.harvard.edu/~apw/stress) for testing the I/O, CPUs, and memory and disks too.

[Elliot B.]

An interesting finding... I attempted to re-install with CentOS 5 for good measure and I received this error about corrupt media during the installation: pasteboard.co/28oIkauh.png -- I hit retry, the install continued without error, I rebooted and the CentOS 5 booted up normally. I never received an error like that with CentOS 6 net-install -- despite re-installing at least a dozen times. Maybe the CentOS 6 net-install isn't catching failures like CentOS 5 does -- do you think that could be the root cause?

[Elliot B.]

I'm familiar with testing memory, hard drive, mobo, etc -- but it's quite time-consuming and I think it's unlikely that two sets of relatively new equipment (differing memory and HDDs) could start having the exact same problem at the exact same time. I had previously installed CentOS on these servers a couple times within the past ~4 months.

[Michael Martinez]

let us continue this discussion in chat

[Jason]

Unfortunately this didn't work for me, I'm now reformatting the entire drive instead, and will install centOS 6.9 yet again.