Grub-install: embedding is not possible in Bios/GPT

Question

I am struggling with setting up an encrypted NAS for some days now. The basic plan is to have btrfs on lvm on luks on raid1 with lvmcache in writeback mode thrown in for the root partition to reduce disk access.

TL;DR:

After setting up the partitions and filesystems GRUB fails to install with:

grub-install: warning: Attempting to install GRUB to a disk with multiple partition labels.  This is not supported yet..
grub-install: error: embedding is not possible, but this is required for RAID and LVM install.

---

Partitions

Following the Arch Wiki I start by setting up the partitions:

gdisk output for /dev/sda and /dev/sdb:

Disk /dev/sda: 976773168 sectors, 465.8 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 9EFA6587-E34F-4AC1-8B56-5262480A6C6A
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 976773134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS boot partition
   2            4096       976773134   465.8 GiB   8300  Linux filesystem

Note the BIOS boot partition that is apparently required by GRUB when installing in BIOS/GPT mode.

MDADM

As I have two disk I want them in a RAID1 array:

mdadm --create --level=1 --raid-devices=2 /dev/md0 /dev/sda2 /dev/sdb2

root@archiso ~ # mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 name=archiso:0 UUID=bdfc3fea:f4a0ee6d:6ac08012:59ea384b

root@archiso ~ # cat /proc/mdstat     
Personalities : [raid1] 
md0 : active raid1 sdb2[1] sda2[0]
      488253440 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  2.0% (9832384/488253440) finish=96.6min speed=82460K/sec
      bitmap: 4/4 pages [16KB], 65536KB chunk

unused devices: <none>

LUKS

Next I setup a LUKS volume on top of the RAID:

root@archiso ~ # cryptsetup luksFormat /dev/md0 

WARNING!
========
This will overwrite data on /dev/md0 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase: 
Verify passphrase:


root@archiso ~ # cryptsetup luksOpen /dev/md0 md0-crypt
Enter passphrase for /dev/md0:

LVM

Btrfs snapshots could be used instead of LVM, but as of writing this there is no way to add a SSD caching device to Btrfs. So I opted to use LVM and add the SSD via lvmcache later:

(Creating the volume group in one step:)

root@archiso ~ # vgcreate vg0 /dev/mapper/md0-crypt 
  Physical volume "/dev/mapper/md0-crypt" successfully created
  Volume group "vg0" successfully created

root@archiso ~ # lvcreate -L 100M -C y vg0 -n boot
  Logical volume "boot" created.
root@archiso ~ # lvcreate -L 20G vg0 -n root
  Logical volume "root" created.
root@archiso ~ # lvcreate -L 10G vg0 -n var
  Logical volume "var" created.
root@archiso ~ # lvcreate -L 6G -C y vg0 -n swap
  Logical volume "swap" created.
root@archiso ~ # lvcreate -l +100%FREE vg0 -n home
  Logical volume "home" created

Resulting in the following layout:

root@archiso ~ # lvs
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  boot vg0  -wc-a----- 100.00m                                                    
  home vg0  -wi-a----- 429.53g                                                    
  root vg0  -wi-a-----  20.00g                                                
  swap vg0  -wc-a-----   6.00g                                                    
  var  vg0  -wi-a-----  10.00g 

Btrfs/Filesystems

Creating the filesystems:

root@archiso ~ # mkfs.ext4 /dev/vg0/boot
root@archiso ~ # mkfs.btrfs /dev/vg0/home
root@archiso ~ # mkfs.btrfs /dev/vg0/root
root@archiso ~ # mkfs.btrfs /dev/vg0/var

(ext4 was chosen for boot because btrfs complains about the small partition size.)

Mounting the filesystems:

root@archiso ~ # swapon /dev/vg0/swap
root@archiso ~ # mount /dev/vg0/root /mnt/arch -o compress=lzo
root@archiso ~ # mount /dev/vg0/home /mnt/arch/home -o compress=lzo
root@archiso ~ # mount /dev/vg0/var /mnt/arch/var -o compress=lzo
root@archiso ~ # mount /dev/vg0/boot /mnt/arch/boot

Installing Arch

Actually I just copy the system from a previous backup:

root@archiso ~ # rsync -Pa /mnt/bkp/sda/* /mnt/arch

(coffee break)

Setting up mdadm.conf and fstab

root@archiso ~ # genfstab -U /mnt/arch > /mnt/arch/etc/fstab
root@archiso ~ # cat /mnt/arch/etc/fstab 
# /dev/mapper/vg0-root
UUID=62ebf0c9-bb37-4b4e-87dd-eb8a4ace6a69       /               btrfs           rw,relatime,compress=lzo,space_cache 0 0

# /dev/mapper/vg0-home
UUID=53113e11-b663-452f-b4da-1443e470b065       /home           btrfs           rw,relatime,compress=lzo,space_cache 0 0

# /dev/mapper/vg0-var
UUID=869ffe10-7a1c-4254-9612-25633c7ae619       /var            btrfs           rw,relatime,compress=lzo,space_cache 0 0

# /dev/mapper/vg0-boot
UUID=d121a9df-8c03-4ad9-a6e0-b68739b1a358       /boot           ext4            rw,relatime,data=ordered        0 2

# /dev/mapper/vg0-swap
UUID=29035eeb-540d-4437-861b-c30597bb7c16       none            swap            defaults        0 0

root@archiso ~ # mdadm --detail --scan >> /mnt/arch/etc/mdadm.conf
root@archiso ~ # cat /mnt/arch/etc/mdadm.conf
[...]
ARRAY /dev/md0 metadata=1.2 name=archiso:0 UUID=bdfc3fea:f4a0ee6d:6ac08012:59ea384b

Chrooting into the system

root@archiso ~ # arch-chroot /mnt/arch /bin/bash
[root@archiso /]#

mkinitcpio.conf

These hooks were added: mdadm_udev encrypt lvm2 btrfs

[root@archiso /]# mkinitcpio -p linux

Configuring GRUB

Now for the interesting (and failing) part, I chose GRUB as my bootloader as it should support all of the contraptions that I use.

References:

• https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system#LVM_on_LUKS
• http://www.pavelkogan.com/2014/05/23/luks-full-disk-encryption/

Changed parts in /etc/default/grub:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md0:vg0"
GRUB_ENABLE_CRYPTODISK=y

Installing grub:

[root@archiso /]# grub-install --target=i386-pc --recheck  /dev/sda                                                  
Installing for i386-pc platform.
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
  /run/lvm/lvmetad.socket: connect failed: No such file or directory
  WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
grub-install: warning: Attempting to install GRUB to a disk with multiple partition labels.  This is not supported yet..
grub-install: error: embedding is not possible, but this is required for RAID and LVM install.

(--debug output available here)

Frankly ... I have no idea what's the problem here. In BIOS/GPT mode GRUB should embed it's core.img into the ef02/BIOS boot partition shouldn't it?

Edit

https://bbs.archlinux.org/viewtopic.php?id=144254 doesn't apply here:

[root@archiso /]# btrfs fi show --all-devices
Label: none  uuid: 62ebf0c9-bb37-4b4e-87dd-eb8a4ace6a69
        Total devices 1 FS bytes used 965.77MiB
        devid    1 size 20.00GiB used 3.04GiB path /dev/mapper/vg0-root

Label: none  uuid: 869ffe10-7a1c-4254-9612-25633c7ae619
        Total devices 1 FS bytes used 339.15MiB
        devid    1 size 10.00GiB used 3.04GiB path /dev/mapper/vg0-var

Label: none  uuid: 53113e11-b663-452f-b4da-1443e470b065
        Total devices 1 FS bytes used 384.00KiB
        devid    1 size 429.53GiB used 2.04GiB path /dev/mapper/vg0-home

Btrfs v3.17.3

Solution

Hmm ... apparently this line was the clue:

grub-install: warning: Attempting to install GRUB to a disk with multiple partition labels.  This is not supported yet..

Previously I had installed btrfs directly on /dev/sda and /dev/sdb. That's why both of them had a FSTYPE and LABEL attached (as shown in lsblk).

Solution: I have now wiped both /dev/sda and /dev/sdb with hdparm (Secure Erase). There is probably a better way to unset those flags ... but this worked for me.

Comments

[derobert]

You could probably use e.g., dd to just wipe the first & maybe last few megabytes. Moot point now, of course.

[fho]

Yep ... thought about that first ... but then found out about the Secure Erase features :)