Grub2 RAID /boot

Question

I'm using Gentoo on my HomeServer. My HDDs are raid1 mirrored with mdadm.

My problem is how to boot the /boot partition correct?

/dev/md1 consists of /dev/sdc2 & /dev/sdd2 mounted on /boot

In my grub.cfg I have the uuid of /dev/md1 but at boot time the array is of course not ready.

simplest way would be to un-raid the /boot partitions, but that is not what I want to do.

so, how do i find out the UUID of the raid members? ls -l /dev/disk/by-uuid only shows me the uuid of /dev/md1. And when I unraid the disks, I cannot mount them to get their UUID mount: unknown filesystem type 'linux_raid_member'

so how can I let Grub2 boot those disks, will it be able to determine the contents before they are in raid? (they are in fact ext2).

I should maybe add that I use gpt partition tables, therefore I have a small partition /dev/sdc1 and /dev/sdd1 on both drives to host the bios_grub partition. those are not raided atm.

As pointed out in the comments below Gilles' answer my problem persists. Grub won't find the device, telling me

error: no such device: 9f81a-(device uuid)-5580.
entering rescue mode...
grub rescue> 

afterwards I added the lines

insmod part_msdos
insmod part_gpt
insmod raid
insmod mdraid

to my grub.cfg, just above the line where root is defined root='(md0)' I still get the same error.

I did some more research and i really guess its an error related to the uuids:

# grub-probe -d --target=fs_uuid /dev/md1
9f81a35d-0813-481f-9ae0-e4fba57c5580
# blkid
/dev/sdc2: UUID="11adb545-0e80-61d1-61f6-565f18e8c3f0" UUID_SUB="88826c5e-d12b-307d-6e54-556d1ebb2458" LABEL="livecd:1" TYPE="linux_raid_member"
/dev/sdd2: UUID="11adb545-0e80-61d1-61f6-565f18e8c3f0" UUID_SUB="15150fb2-5066-edba-d39b-08b63219453c" LABEL="livecd:1" TYPE="linux_raid_member"
/dev/md1: UUID="9f81a35d-0813-481f-9ae0-e4fba57c5580" TYPE="ext2"
(I cut away the other drives here)

as you can see both partitions have the same UUID, so I edited the UUID of the drives in my grub.cfg and did grub-install /dev/sdx again, but I still get the error with the "old" /dev/md1 UUID.

Solution

The Gentoo Wiki has information that discusses both Grub 1 and Grub 2. Note that the situation is completely different in Grub 1 and Grub 2: Grub 1 doesn't know anything about RAID, but can fake reading from a RAID-1 device by reading one of the disks (this requires a 0.9 or 1.0 mdraid superblock format, not 1.1 or 1.2); Grub 2 supports Linux mdraid volumes natively.

Grub 2 works in three stages (see here for more details):

1. The boot sector.
2. A core image, generated by install-grub in /boot/grub/core.img, capable of showing the rescue prompt and loading modules but not much else.
3. A full system with loadable modules, which normally (through the normal module) shows a boot menu described by /boot/grub/grub.cfg.

Generating the core image

Run grub-install /dev/sda to generate the core image and populate /boot/grub with module files. grub-install also installs a boot sector on the indicated device. Run both grub-install /dev/sdc and grub-install /dev/sdd if you want to have a boot sector on both drives.

If grub-install doesn't detect your drives properly, create the following script as /usr/local/sbin/grub-probe-verbose and run grub-install --grub-probe=/usr/local/sbin/grub-probe-verbose to see what's going on.

#!/bin/sh
exec /usr/sbin/grub-probe "$@" | tee /dev/stderr

From your transcript from bash -x /sbin/grub-install /dev/sdc, it appears that grub-probe isn't detecting /dev/md1 as a RAID array (/sbin/grub-probe --device-map=/boot/grub/device.map --target=partmap --device /dev/md1 and /sbin/grub-probe --device-map=/boot/grub/device.map --target=abstraction --device /dev/md1 return nothing). In case the issue is only in grub-probe and not in the boot code, try overriding its decision:

grub-install --modules='biosdisk ext2 msdos raid mdraid' /dev/sdc

Support for mdraid 1.x superblocks is still recent; it's not in Grub 1.98, you need at least Bazaar revision 2550 dated 2010-07-20 (the code was on a branch for a few months before). Debian squeeze and Ubuntu 10.10 ship with 1.x superblock support; I would expect Gentoo to have 1.99 by now.

Generating grub.cfg

Run grub-mkconfig -o /boot/grub/grub.cfg.grub-mkconfig to produce a tentative Grub configuration file. Review it, and if looks correct, rename it to /boot/grub/grub.cfg.

Device map

You may need to tweak /boot/grub/device.map to get the output of grub-mkconfig right. This file will be generated automatically by grub-mkdevicemap if it doesn't exist, but setups with many disks tend to confuse it. I would expect it to contain something like

(md0) /dev/md1
(hd0) /dev/sdc
(hd1) /dev/sdd

where md0 is the Grub volume name and /dev/md1 is the name under Linux. Grub needs to access /boot/grub for two things:

• Grub's boot sector reads the core image at a fixed location on a disk supported by the BIOS (it doesn't have much choice). With a PC BIOS, (hd0) is the disk that the BIOS reads the boot sector from (it's the disk that contains the boot sector), and (hd1) is some other disk (you don't always get to choose). It's simpler if (hd0) is the disk (or a disk, with a RAID-1 array) that contains /boot/grub.
• The core image loads modules and grub.cfg and the Linux kernel from a filesystem location determined by grub-install, typically /grub or /boot/grub from a device that can be a RAID array or any other volume type supported by Grub.

If you change device.map, you need to run grub-install again. You may need to run grub-mkconfig again if you're not relying on the search command for everything.

Comments

[Gilles 'SO- stop being evil']

grub.cfg isn't involved at this point: grub-install doesn't use it, and at boot time Grub gets stuck before loading it. Apparently your core image is missing RAID support. What superblock format version does your array have? And please post a transcript of bash -x /usr/sbin/grub-install /dev/sdc.

[Coffeee]

pastebin.com/JRrWfyBM here we go. superblock on all arrays is 1.2 .

[Alen Milakovic]

Perhaps I am missing something, but you should be able to generate the grub config automatically with grub-install and update-grub. Writing it manually is usually not necessary. Does that not work for you?

[Coffeee]

nope, for some setups it does not work out. changing the grub.cfg is maybe not the best way, you could also edit the files from which grub-mkconfig gets it information but in the end there is very little difference. At the moment I just boot from a USB-stick where grub and kernel are stored on, not what i initially wanted to do, but pretty nice, too.

[Coffeee]

The Link looks promising. I already modified my device.map because I had a device not found error (/dev/md1), but i in fact did not think of using grub-counting-style.

[Coffeee]

I added the line (md0) /dev/md1 to my device.map and grub-mkconfig found the corresponding UUID and wrote it to the grub.cfg. but after booting I get this error: no such device: 9f81a...restofUUID

[Gilles 'SO- stop being evil']

@WalterMaier-Murdnelch Do you have the necessary insmod directives (insmod part_msdos, insmod raid, insmod mdraid should do it)? I would have expected grub-mkconfig to generate them, but maybe it didn't. Post your grub.cfg.

[Coffeee]

pastebin.com/qWaSYw8a the insmod stuff is missing, thats exactly what I need I guess, will try it right away

[Coffeee]

still the same error... strange.

[psusi]

Grub2 prefers to use grub-probe to figure things out automagically instead of using a static device.map. It can boot just fine without a separate /boot partition also. That should also allow grub-mkconfig to automatically build the required raid modules into the core image.

[Coffeee]

the separate /boot is necessary, rest of the drive is encrypted... but lets not discuss this here, I think I will get some further problems with that as well. and yeah i read somewhere that i can also boot an encrypted partition with grub, but atm i cant even boot an unencrpyted one, so lets keep it simple ;)

[Coffeee]

i changed my device.map according to your recommendations. i still get dropped to the rescue prompt. entering commands like insmod (md0,1)/ yields no such disk, insmod (hd0,1)/ yields no such partition and insmod (hd0)/ yields unknown filesystem. where can I specify which modules are built into the core.img?

[Coffeee]

i just read here linuxquestions.org/questions/linux-general-1/… that grub gets its raid info from /etc/mdadm/mdadm.conf

[Coffeee]

entering mdadm --auto-detect in the rescue console does not work, but entering ls yields (hd0) (hd1) (hd2) (hd3) (fd0)

[Coffeee]

sinodun.com/2010/07/installing-grub2-on-gnulinux-software-ra‌​id this blog-entry looks promising as well, working through it atm.

[psusi]

grub doesn't care about mdadm.conf; it reads the raid superblocks on the disk. You can't load a module from the disk when the module you are missing is one required to access the disk. Normally grub-install runs grub-probe which tells it that /boot is on a raid, so it builds raid module into the core image. This doesn't seem to be happening for you.

[Gilles 'SO- stop being evil']

@WalterMaier-Murdnelch grub-probe doesn't detect your RAID array as such. Make sure you have a recent enough Grub, and try forcing the necessary modules into core.img (see my edit).

[Coffeee]

thanks so much, I added the modules to the grub-install, now i am dropped out to the "normal" grub console. bad thing is there is no error output, so I am not sure whats the error. :/ edit: my grub version is 1.99~rc1

[Mat]

Please don't just post dumps of commands, it's generally not sufficient. Add explanations. And make sure what you do dump is actually relevant to the question.

[Coffeee]

if it were this easy Gilles and me would have done it, I tried it again with a completely other computer and still didn't got it working. and btw is this answer using grub legacy not 2