boot issues - long delay, then "gave up waiting for root device"

Question

I've had this issue on and off for about two years now. I noticed it on a new (custom built) machine running 10.04 when that first came out, but then it went away until a few months ago. I've gone through a number of hard drive changes but I can't say specifically what if anything I changed hardware-wise to make it stop or start happening. I had assumed upgrading to a modern Ubuntu version would fix the issue, so I installed 12.04 beta on a spare partition last night, but it's still happening.

Here's the issue. After grub loads and I select a kernel to boot, the screen goes blank save for a blinking cursor. It sits in this state for many long minutes before it finally gives up and gives me an initramfs shell with the message gave up waiting for root device (and lists the /dev/disk/by-uuid/... path it was waiting for) but no other specific diagnostic information.

Now, here's the tricky part. For one, the problem is intermittent - sometimes it progresses from the blinking cursor to the Ubuntu splash boot screen in a few seconds, and once it gets that far it always continues booting fine. The really bizarre thing is that I can "force" it to "find" the root device by repeatedly pressing the space bar and hitting the machine's power button. If I tap those enough, eventually I will notice the hard drive light coming on, at which point it will always continue the boot process after a few seconds. Interestingly, if I wait slightly too long before pressing the power button (30s?), as soon as I press it I get the gave up waiting message and the initramfs shell.

I've tried setting up /etc/fstab (and the grub menu.lst or whatever it's called nowadays) to use device names (e.g. /dev/sda1) instead of UUIDs, but I get the same effect just with the device name, not UUID, in the error message.

I should also mention that when I boot to Windows 7, there is no issue. It boots slowly all the time just by virtue of being Windows, but it never hangs indefinitely. This would seem to indicate it's a problem in Ubuntu, not the hardware.

It's pretty annoying to have to babysit the computer every time it boots. Any ideas? I'm at a loss. Not even sure how to diagnose the issue. Thanks!

EDIT:

Here's some dmesg output from 10.04. The 15 second gap is where it was doing nothing. I pressed the power button and space bar a few times, and the stuff at 16 seconds happened. Not sure what any of it means.

[    1.320250] scsi18 : ahci
[    1.320294] scsi19 : ahci
[    1.320320] ata19: SATA max UDMA/133 abar m8192@0xfd4fe000 port 0xfd4fe100 ir
q 18
[    1.320323] ata20: SATA max UDMA/133 abar m8192@0xfd4fe000 port 0xfd4fe180 ir
q 18
[    1.403886] usb 2-4: new high speed USB device using ehci_hcd and address 4
[    1.562558] usb 2-4: configuration #1 chosen from 1 choice
[   16.477824] ata16: SATA link down (SStatus 0 SControl 300)
[   16.477843] ata19: SATA link down (SStatus 0 SControl 300)
[   16.477857] ata3: SATA link down (SStatus 0 SControl 300)
[   16.477895] ata15: SATA link down (SStatus 0 SControl 300)
[   16.477906] ata20: SATA link down (SStatus 0 SControl 300)
[   16.477977] ata17: SATA link down (SStatus 0 SControl 300)
[   16.478003] ata12: SATA link down (SStatus 0 SControl 300)
[   16.478046] ata13: SATA link down (SStatus 0 SControl 300)
[   16.478063] ata14: SATA link down (SStatus 0 SControl 300)
[   16.478108] ata11: SATA link down (SStatus 0 SControl 300)
[   16.478123] ata18: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[   16.478127] ata6: SATA link down (SStatus 0 SControl 300)
[   16.478157] ata5: SATA link down (SStatus 0 SControl 300)
[   16.478193] ata18.00: ATAPI: MARVELL VIRTUALL, 1.09, max UDMA/66

After that, it took its sweet time, and I had to keep hitting space bar to coax it along. Here's some more dmesg output from a little later in the boot process:

[   17.982291] input: BTC USB Multimedia Keyboard as /devices/pci0000:00/0000:00
:13.0/usb5/5-2/5-2:1.0/input/input4
[   17.982335] generic-usb 0003:046E:5506.0002: input,hidraw1: USB HID v1.10 Key
board [BTC USB Multimedia Keyboard] on usb-0000:00:13.0-2/input0
[   18.005211] input: BTC USB Multimedia Keyboard as /devices/pci0000:00/0000:00
:13.0/usb5/5-2/5-2:1.1/input/input5
[   18.005274] generic-usb 0003:046E:5506.0003: input,hiddev96,hidraw2: USB HID 
v1.10 Device [BTC USB Multimedia Keyboard] on usb-0000:00:13.0-2/input1
[   22.484906] EXT4-fs (sda6): INFO: recovery required on readonly filesystem
[   22.484910] EXT4-fs (sda6): write access will be enabled during recovery
[   22.548542] EXT4-fs (sda6): recovery complete
[   22.549074] EXT4-fs (sda6): mounted filesystem with ordered data mode
[   32.516772] Adding 20482832k swap on /dev/sda5.  Priority:-1 extents:1 across:20482832k 
[   32.742540] udev: starting version 151
[   33.002004] Bluetooth: Atheros AR30xx firmware driver ver 1.0
[   33.008135] parport_pc 00:09: reported by Plug and Play ACPI
[   33.008186] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[   33.012076] lp: driver loaded but no devices found
[   33.037271] ppdev: user-space parallel port driver
[   33.090256] lp0: using parport0 (interrupt-driven).

Any clues in there?

FURTHER EDIT:

Some screenshots of dmesg output from the initramfs busybox shell when it fails:

I tried fossfreedom's suggestion of trying a number boot options. So far, it looks like irqpoll is actually working! I've booted four times successfully with it so far.

(As a side note, I can't get irqpoll to stick -- I edited /etc/default/grub to add it to GRUB_CMDLINE_LINUX_DEFAULT and ran update-grub, but still nothin'. What am I missing?) (My grub was hosed somehow when installing the other ubuntus. I fixed it and am seeing irqpoll always now.)

Given that, what is the problem here? It sounds like irqpoll is a hack to get BIOSes that somehow don't adhere to some IRQ standard to work with Linux. How do I figure out what component is the issue, and is there another solution to get Linux to play nice with it that doesn't involve what sounds like so much overhead?

Solution

Your latest dmesg messages look interesting.

I've seen similar press space type problems during booting on one laptop I owned.

The solution I found was to add to grub the noapic or the nolapic grub boot parameter.

There are a number of grub boot options you can try as per the community wiki:

https://help.ubuntu.com/community/BootOptions

Thus if standard workaround grub boot options such as noapic or acpi=off dont work for you, some of the other common options could work for you.

When testing grub boot option you could use my answer here to temporarily add the kernel options at boot time.

How do I set 'nomodeset' after I've already installed Ubuntu?

---

irqpoll:

from the wiki: Changes the way the kernel handles interrupt calls (set it to polling). Can be useful in case of hardware interrupt issues.

  irqpoll         [HW]
                        When an interrupt is not handled search all handlers
                        for it. Also check all handlers each timer
                        interrupt. Intended to get systems with badly broken
                        firmware running

source

Thus - and since you have a custom built computer - it is very likely to be one or more components that make up your computer. To discover which one will require a bit of detective work.

The easiest way would be just to simplify you computer to the basics - use a fixed keyboard, simple mouse, onboard graphics, onboard ethernet and nothing else. i.e. pull out all the PCI cards/USB components you are using. Then add each one back in one at a time.

If you find that you still need irqpoll on a simplified build - then you'll need to take a closer look at your motherboard. Look at the manufacturer site and see if there are any firmware updates - most probably you'll need to dos-boot to flash these.

Comments

[nilsonneto]

have you tried any kernel boot options such as nomodeset and/or acpi=off? - use this Q&A to guide you: askubuntu.com/questions/38780/…

[psusi]

Check dmesg for errors when it fails to find the disk and drops you to the busybox promt, not when it manages to work.

[chazomaticus]

Added some screenshots of dmesg output when it fails. Also discovered kernel boot option irqpoll to work to fix things. See latest edit. Thanks, guys!

[nilsonneto]

please can you pastebin.com your /etc/default/grub and /boot/grub/grub.cfg files please

[chazomaticus]

Turns out my comment about grub not being updated was due to it being somehow hosed from installing two other ubuntus. I guess if I have a /boot partition they all need to share it. Anyway, that's no longer an issue. Thanks for your help!

[chazomaticus]

I will try this when I get home. What possible reason would tapping the space bar and/or power button have for making this mount suddenly succeed? Also wouldn't that mean I no longer have access to a device once booted? ALSO did you mean /etc/fstab instead of /etc/sstab?

[chazomaticus]

Actually, now that I'm re-reading your answer, it's just not a solution to my problem. I want everything listed in /etc/fstab to be mounted. That's kind of why they're in there. Is there an answer that doesn't involve just not mounting a hard drive at all?

[Ringtail]

mount it after boot if that is what you want. I was trying to resolve your issue "during" boot. If that's the case????

[chazomaticus]

What UUID am I removing from fstab? Am I to just try them at random until I find the one that's causing the problem? What if it's my root partition? Or home?

[Ringtail]

did you try the suggestion about pressing "s" to skip the mount while booting?

[chazomaticus]

The UUID it's listing when it fails is the root device, hence "gave up waiting for root device". I need that to boot. I didn't try pressing S (by the way, what does "when booting" mean? During grub? While the screen is blank with the blinking cursor?) because I don't believe your answer is a viable solution to my problem.

[psusi]

Pressing S to skip applies to ancillary mounts that happen later in the boot process, not the root fs, which lands you in the busybox prompt as described when it can't be found.

[chazomaticus]

Thanks for the suggestion! irqpoll seems to do the trick (booted successfully four times in a row!), but I can't make it stick. How do I add it to the kernel line permanently? Also, is there a way to figure out the specific issue and use something more specific than irqpoll, which sounds like something of a firehose? See my edit for more.

[nilsonneto]

please see my update - also - I've dropped a comment asking for more info about your grub. thanks.