System unbootable - grub error: disk lvmid not found

Question

I'm gettind this error at boot:

grub error: disk 'lvid/caoMWu-o417-GMgh-6vFj-1qrw-iJMi-ypwm0f/Z2eotR-N0HN-nrol-3hUd-odMB-GzHy-4PrsnL' not found. Entering rescue mode..

I can't even get the GRUB menu, the error message show immediately up.

Now here's the situation: I had to replace my motherboard because it was broken, and just after the change (I've replaced with an identical one) I'm not able to boot anymore.

This is my setup: Archlinux distro, two hdd with a RAID1 and LVM builded on top of it.

Now I can't understand what happened, cause the UUID of the disk/partition should be remained the same even after mb replacement ?

What can I do/check to restore the system ?

Below some info about the LVM setup:

--- Volume group ---
  VG Name               server
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               2.73 TiB
  PE Size               4.00 MiB
  Total PE              715318
  Alloc PE / Size       715318 / 2.73 TiB
  Free  PE / Size       0 / 0   
  VG UUID               caoMWu-o417-GMgH-6vFj-1qrv-iJMi-ypwm0f


  --- Logical volume ---
  LV Path                /dev/server/boot
  LV Name                boot
  VG Name                server
  LV UUID                3Z70U8-Mc1W-aKQU-tORg-oiaT-P2kI-aYG74K
  LV Write Access        read/write
  LV Creation host, time archiso, 2015-11-06 08:51:19 +0100
  LV Status              available
  # open                 1
  LV Size                512.00 MiB
  Current LE             128
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/server/swap
  LV Name                swap
  VG Name                server
  LV UUID                z8PdBc-DZRL-zsC5-190M-nyUl-k3cn-9gaYRo
  LV Write Access        read/write
  LV Creation host, time archiso, 2015-11-06 08:51:28 +0100
  LV Status              available
  # open                 0
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Path                /dev/server/archroot
  LV Name                archroot
  VG Name                server
  LV UUID                Z2eotR-N0HN-nroI-3hUd-odMB-GzHy-4PrsnL
  LV Write Access        read/write
  LV Creation host, time archiso, 2015-11-06 08:52:37 +0100
  LV Status              available
  # open                 1
  LV Size                20.00 GiB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/server/storage
  LV Name                storage
  VG Name                server
  LV UUID                cccOsD-GBXu-pMws-e4bS-tCic-ZCFs-OstIf9
  LV Write Access        read/write
  LV Creation host, time archiso, 2015-11-06 08:52:52 +0100
  LV Status              available
  # open                 0
  LV Size                2.70 TiB
  Current LE             709046
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

and the /etc/fstab :

# /dev/mapper/server-archroot
UUID=c83b58a5-d4fd-4634-b6b7-1726ca6ffeb6   /           ext4        rw,relatime,data=ordered    0 1

# /dev/mapper/server-boot
UUID=1fffa9ae-ea36-4888-870e-47bd8fc16268   /boot       ext4        rw,relatime,data=ordered    0 2

# /dev/mapper/server-swap
UUID=a787eefb-692d-4d2a-9482-11196967ea62   none        swap        defaults    0 0

# /dev/mapper/server-storage
UUID=ae082954-b90a-482a-b002-e53f6f75df5a   /mnt/storage    ext4        users,rw,relatime,auto,exec,data=ordered

Solution

You will get this error if you're trying to have GRUB unlock a LUKS2 encrypted volume on boot.

As of Dec 2020, released versions of GRUB only support LUKS1 headers, though LUKS2 support has been added upstream.

Comments

[Brian Stinar]

So what happened? What did you do to get this working? The below solutions didn't work for me.

[lorenzolorenzo]

Actually all was fine. I was running a HP microserver G8 wich has hardware RAID system wich is not compatible with linux (as fair as i understand). So basically I just disabled it from bios, like already did with the old motherboard.

[lorenzolorenzo]

I've already tried it from a live image. Both update-grub and mkinitcpio are exectuted without errors, but didn't solve the problem.

[pevik]

I wander, what was the cause. I had a same problem, but grub2-install (which is run by update-grub) reported errors as the partition was really broken and even cryptsetup luksOpen /dev/sdXY crypt didn't work.

[dannysauer]

While this is good information, it's not relevant to this specific question. If you look at the "disk not found" message and then the vgdisplay and lvdisplay output shown in the question, they match. More importantly, the UUIDs in /dev/disk-by-uuid are the filesystem UUID, not the LVM vg and lv uuids necessary in the lvmid path grub is looking for.

[Nasir Riley]

@dannysauer Where do you see that the UUID in the error message matches those for the LVs? They look different to me. It's possible that they do match the filesystem IDs but we don't know because they were never included in the question. I'm referring to the filesystem UUID which is what that error means but again, we don't know, which is why I directed him to find what they are and make sure that they match. In any event, he seems to have fixed it anyway.

[dannysauer]

Yeah, I see the confusion. When grub boots from lvm, the device is specified as lvmid/<vg uuid>/<lv uuid> instead of the (hd0,1) you'd use for a partition. The root device specified matches the lvm output. :) I just wanted to clarify for someone who finds this later. :)

[pevik]

Yes. But fortunately there is already LUKS2 implementation in current upstream git. I'm playing with it.

[kynan]

That's right, but it hasn't made it into any distro yet afaik, not even Arch.

[Gaurav Mall]

It seems GRUB release schedule is very sparse (1 release every two years, maybe more). However, a new release should've happened December 2020, so it seems we are not far from it being officially supported on a stable release.

[Dementured Old Grump]

Oops again. Reading further down ...... I see the incantations issued in Recovery Mode. Never been able to do anything in this mode. Perhaps I should read the man page. Demetured Old Grump

[Jeremy Boden]

For SATA drives, just removing one end of the data cable is sufficient to disconnect them.