Does GRUB2 support /boot on LVM on MD-raid?

Question

What are the known supported (and unsupported) features for the GRUB2 lvm module in Debian 9? E.g. does it support the current metadata format "lvm2"?

Is there some extra command or configuration required to access LVM logical volumes from GRUB, even after loading the GRUB lvm module?

GRUB appears to support MD-raid, but what about when LVM is running on top of MD-raid, does it affect the above questions somehow? Is grub2-mkconfig expected to support /boot on MD+LVM?

Context

I need to migrate files from one existing disk into the RAID, so I'm not using the installer to set things up. This is for Debian Stretch. The general idea is

1. copy your existing system to a degraded RAID on the new disk
2. boot the RAID system
3. very carefully double-check and then have the RAID swallow the old disk.

I'm aware this isn't necessarily the most sensible of ideas.

I'm looking at booting from LVM on MD-raid. I won't bother faffing with /boot as a separate MD-raid partition, if I don't need to. It looked like this is what the Debian installer was being used to do in https://wiki.debian.org/DebianInstaller/SoftwareRaidRoot

I created an LV on top of MD-raid, and copied the root filesystem to it. But unlike the pure MD-raid setup described on the RAID wiki, grub-mkconfig did not find the new Linux instance.

I booted to the GRUB prompt of the original Linux instance, to see what it could do. I can load the part_gpt, mdraid1x, and lvm modules. There is no lvm2 module. This successfully found devices for my GPT partitions, and the MD device (md/linux) I had created. But it did not find any of the LVM devices I had created, which should have looked like (lvm/vg_linux/lv_root).

The probe command does not work at all on either (hd1,gpt1) or (md/linux); it just says error: unknown filesystem. So it's not capable of identifying the MD-raid, let alone the LVM stuff, even though GRUB is already accessing the MD-raid.

Solution

GRUB2 on Debian 9 is definitely able to read from a normal ("linear") LVM2 volume on MD raid1 (metadata v1.2). However I found this only seems to work if you do insmod lvm before insmod mdraid1x.

If I load them in the opposite order, it changes the order of the devices shown by the GRUB ls command (!), but it does not add the lvm devices to the list.

Standard automatic configuration

To get grub-mkconfig (which is the backend for Debian's update-grub) to detect the second Linux instance as suggested on the RAID wiki page, it sufficed to install the os-prober package. (I expect the Debian installer would have installed this package automatically, if it had seen a second OS when it originally installed the boot loader. Particularly if there was a Windows install).

Interestingly, the generated config loaded lvm after mdraid1x. However it also loaded diskfilter before mdraid1x. It also used the hidden lvmid/... path instead of (lvm/...). For the sake of our sanity, let us ignore the latter difference, and imagine that the early diskfilter load is probably what lets this config work :).

I'm not sure where you're expected to learn either of these methods :-/.

Comments

[Alen Milakovic]

I use LVM on top of MDRAID. I've never had to do any manual work to get this working with GRUB. I've just run update-grub and it seems to be able to figure out what modules to load. Is there something I'm missing?

[Alen Milakovic]

I see. I recently migrated an non-UEFI system to UEFI, and it was no picnic. So I can relate to what you are going through. Having said that, your strategy might be sub-optimal. Asking about how to migrate this setup (if you haven't already done so) would make a good question, I think.

[user2948306]

@FaheemMitha this strategy seems to be fairly well-known for migrating to RAID. I would agree with anyone who said this QA ended up with a lot of irrelevant detail: the root cause turned out to be nothing to do with LVM. (I didn't think I'd find this answer myself. I was expecting to give up and create a second RAID for /boot, which makes for a sub-optimally complicated process IMO).

[user2948306]

The most straightforward way to achieve my goal here would be to throw a little more resource at it, i.e. backup to a third disk. Then I could rely on the installer RAID+LVM support. But... that will leave me with yet more hardware floating around. My goal is to put some old hardware to use for offline backups, and I will be rearranging other hardware which provides online backups. Without putting too much of this redundancy at risk in any one transition.

[user2948306]

I'm sure I spend more time on this than is reasonable. But I'm getting very confident with these tools. It looks like the os-prober thing was the only part I wasn't really familiar with. Fortunately the target hardware is MBR, and I can test this very nicely in a VM before touching the real system. Getting hold of EFI firmware for VMs seems to be more awkward due to licensing restrictions.

[Alen Milakovic]

I agree (in case there is any doubt) that GRUB and friends are very under-documented and quite likely buggy as hell. But that is probably par for the course for a lot of free software.

[Alen Milakovic]

Looking at your comments, I see you wrote "because booting a different OS requires installing os-prober". But I don't understand what that means. What different OS?

[Alen Milakovic]

I see. So if you have more than one instance of Linux, GRUB requires os-prober in order to generate entries for both of them? And you didn't have that installed?

[user2948306]

@FaheemMitha that's my understanding. If you have a suggestion how to make this point more obvious in my answer, let me know :). I've edited both Q&A a little bit, towards that goal.

[Alen Milakovic]

os-prober is a Recommends for grub-common, so you really should have had it installed. (Brought to my attention by aptitide why).

[user2948306]

@FaheemMitha The Jessie live installer disagreed. I don't expect the live makes a difference. See the relevant part of the log from the installer. I expect the installer gainsays the package on the basis that it has more information. I agree that this feels very icky. Disk space is cheap...

[Alen Milakovic]

Interesting. Sounds like possible material for a bug report. Though of course this might have changed since the last release.

[user3299406]

I have had the same experience with Ubuntu 16.04, the result depends on the order of the insmod commands. Thank you for sharing your solution!