CentOS7 Dracut lvm command not found

Question

I updated CentOS7 kernel because of Meltdown issue from 3.10.0-693.11.1 to 3.10.0-693.11.6, but after that update, I cannot boot as centos drop to the dracut with error

warning /dev/mapper/centos-root does not exist

dracut:/# ls /dev/mapper/
control
dracut:/# 

It looks like my initrd is wrongly configured, I can still boot 3.10.0-693.11.1 without any issue. I am trying to debug this (there are a lot resources online) but it looks that I am missing something fundamentally here as I cannot execute lvm command

dracut:/# lvm scan
sh: lvm: command not found
dracut:/# ls /etc/lvm/lvm.conf
ls: cannot access /etc/lvm/lvm.conf: No file or dictionary
dracut:/# ls /etc/lvm
ls: cannot access /etc/lvm: No file or dictionary

I saw that usually problem with tools is caused by missing binaries in /sbin, so I botted stabile kernel and checked:

root@localhost ~# ls /sbin/lvm
/sbin/lvm
root@localhost ~# which lvm
/sbin/lvm

So binary is at correct place, but for some reason dracut is not seeing it inside /sbin.

My grub configuration for 3.10.0-693.11.1 and 3.10.0-693.11.6 is almost identical

#: after update kernel params
linux16 /vmlinuz-3.10.0-693.11.6.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=128M rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rdshell 
#: before update kernel params
linux16 /vmlinuz-3.10.0-693.11.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=128M rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rdshell

Here I struggled a little bit: I went through dracut man page and there a lot of options for LVM, I pressume autodiscovery is not working but why it works for previous kernel and newer has an issue? Also how I can get working lvm tools on initram, to debug it further?

Thank you for any advice!

UPDATE: Running lsinitrd -m -k show that there is no lvm module inside ramdisk

========================================================================
Image: /boot/initramfs-3.10.0-693.11.6.el7.x86_64.img: 18M
========================================================================
Early CPIO image
========================================================================
drwxr-xr-x   3 root     root            0 Jan 22 11:15 .
-rw-r--r--   1 root     root            2 Jan 22 11:15 early_cpio
drwxr-xr-x   3 root     root            0 Jan 22 11:15 kernel
drwxr-xr-x   3 root     root            0 Jan 22 11:15 kernel/x86
drwxr-xr-x   2 root     root            0 Jan 22 11:15 kernel/x86    /microcode
-rw-r--r--   1 root     root        24576 Jan 22 11:15 kernel/x86    /microcode/GenuineIntel.bin
========================================================================
Version: dracut-033-502.el7

dracut modules:
bash
nss-softokn
i18n
network
ifcfg
drm
plymouth
kernel-modules
qemu
resume
rootfs-block
terminfo
udev-rules
biosdevname
systemd
usrmount
base
fs-lib
shutdown
========================================================================

I was trying to add lvm kernel driver to initram image but unfortuneatly that failed without any visible issue:

dracut --add-drivers  lvm /boot/initramfs-3.10.0-693.11.6.el7.x86_64.img

dracut[27939]: Failed to install module lvm

adding -v doesn't point to anything usefull.

Solution

I had this problem. I fixed it by:

logging into a live cd (recovery should be fine too)

cp /etc/dracut.conf /etc/dracut.conf.bak

nano /etc/dracut.conf

Ammending the following lines to:

# dracut modules to add to the default
add_dracutmodules+="lvm"

# install local /etc/mdadm.conf
mdadmconf="yes"

# install local /etc/lvm/lvm.conf
lvmconf="yes"

(You probably won't need mdadmconf, I added it as I have Raid.)

Rerun dracut -f for my kernel. Reboot.

This got me passed the lvm issue. I now have issues sysroot not mounting (it drops me into a recovery shell). I will update when fixed.

Comments

[Michael Hampton]

The output of for version in $(rpm -q --qf %{VERSION}-%{RELEASE}.%{ARCH}\\n kernel); do lsinitrd -m -k $version; done will show the contents of the initial ramdisk for each installed kernel version. It should contain the lvm module (among others).

[user979291]

Thanks @MichaelHampton for reply I just checked and in new module, there is no lvm inside ramdisk, I will try to add it and rerun

[user979291]

Thanks for help with dracut I can move forward with my debugging, no idea why it wasn't added at the first place?

[blur]

Vau, this helped in my case, why it has only 0 votes???