Ubuntu 16.04 stops booting, it has LVM and LUKS encryption configured, superblock restore already tried

Question

Suddenly my Ubuntu 16.04 drops me into an initramfs prompt like this:

BusyBox v.1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)   
Enter 'help' for list of built-in commands.  

(initramfs)

If I type exit I get:

Gave up waiting for root device. Common problems:
  — Boot args (cat /proc/cmdline)
    — Check rootdelay= (did the system wait long enough?)
    — Check root= (did the system wait for the right device?)
  — Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/ubuntu--vg-root does not exist. Dropping to a shell!

BusyBox v.1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)   
Enter 'help' for list of built-in commands. 

(initramfs)

Things I've tried, without success so far:

• Tried recovering a superblock backup with a USB boot disk, still boots into this.
• Cannot boot into a different kernel because when pressing SHIFT at grub time I only get one kernel in the advanced options.

When I get at the grub screen, if I press e these are my settings:

setparams 'Ubuntu'

    recordfail
    load_video
    gfxmode $linux_gfx_mode
    insmod gzio
    if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
    insmod part_msdos
    insmod ext2
    set root='hd0,msdos1'
    if [ x$feature_platform_search_hint = xy ]; then
      search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 7ec61194-f216-4f1b-b684-5b2e2d643c25
    else
      search --no-floppy --fs-uuid --set=root 7ec61194-f216-4f1b-b684-5b2e2d643c25
    fi
    linux        /vmlinux-4.4.0-24-generic root=/dev/mapper/ubuntu--vg-root ro  quiet splash $vt_handoff
    initrd         /initrd.img-4.4.0-24-generic

If I boot into a bootable LIVE usb disk sudo fdisk -l returns:

Disk /dev/sda: 480.1 GB, 480103981056 bytes
255 heads, 63 sectors/track, 58369 cylinders, total 937703088 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x60229a0a

   Device  Boot     Start       End     Blocks    Id   System
/dev/sda1    *       2048    999423     498688    83   Linux
/dev/sda2         1001470 937701375  468349953     5   Extended
/dev/sda5         1001472 937701375  468349952    83   Linux

Disk /dev/sdb: (skipping since I think this is the USB disk...)

If I mount the partitions via nautilus (the encrypted one prompts me for my password, which I enter), then sudo lsblk returns:

NAME                                                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                                    8:0    0 447.1G  0 disk
├─sda1                                                 8:1    0   487M  0 part /media/ubuntu/7ec61194-f216-4f1b-b684-5b2e2d643c
├─sda2                                                 8:2    0     1K  0 part
└─sda5                                                 8:5    0 446.7G  0 part
  └─luks-11613ca4-65f2-4079-a750-6f71b6318903 (dm-0) 252:0    0 446.7G  0 crypt
    └─ubuntu--vg-root (dm-1)                         252:1    0 439.2G  0 lvm
    └─ubuntu--vg-swap_1 (dm-2)                       252:2    0   7.5G  0 lvm
sdb                                                    8:16   1   1.9G  0 disk
└─sdb1                                                 8:17   1   1.9G  0 part /cdrom
sr0                                                   11:0    1  1024M  0 rom
loop0                                                  7:0    0 938.7M  1 loop

What's wrong? What is preventing the system to boot up and ask me the encryption password like usual? Thanks!

Solution

You can try running the script below from the LiveCD (possibly updated versions at the link). The script attempts to partially automate the process for you. It was written and tested for an NVMe drive, so you may need to change things slightly if you are using a SATA/IDE drive with an /dev/sd* path. If you aren't using EFI you can delete any line containing efi or EFI by running sed -i -e '/efi/d' -e '/EFI/d' crypt-fix.sh after saving the script to your computer. Will have to add some detection logic for that eventually.

https://gist.github.com/dragon788/e777ba64d373210e4f6306ad40ee0e80

crypt-fix.sh

#!/bin/bash
# Call with `sudo bash DEBUG=1 ./crypt-fix.sh` for verbose output
[ -n "$DEBUG"] && set -x
# Prompt user for device from /dev/sd* /dev/nvme* /dev/mmc* prefixes?
# For /dev/sda probably sda1 is EFI and sda2 is boot and sda3 is encrypted
DEVICE=/dev/nvme0n1
EFIPATH="${DEVICE}p1"
BOOTPATH="${DEVICE}p2"
CRYPTPATH="${DEVICE}p3"
TARGETPATH=/mnt
# Need root for mounting stuff
if ! (( $EUID == 0 )); then echo "Please run with `sudo $0`"; fi

clear_mounts () {
# Clears mounts in case of interrupt or upon exit to allow running script multiple times
umount $TARGETPATH/boot/efi
umount $TARGETPATH/boot
umount $TARGETPATH/proc
umount $TARGETPATH/dev
umount $TARGETPATH
vgchange -an
cryptsetup close temp_name
cryptsetup close $CRYPTNAME
set +x
}
trap clear_mounts INT EXIT

cryptsetup open $CRYPTPATH temp_name
vgchange -ay
# Can't get this until LVM devices are scanned above
ROOTPATH=$(ls /dev/mapper/* | grep root)
# Make sure nothing else is mounted on our $TARGETPATH
umount $TARGETPATH
wait
mount $ROOTPATH $TARGETPATH
# Find the name that is required for `update-initramfs` to properly update things
CRYPTNAME=$(cat $TARGETPATH/etc/crypttab | awk '/^[ ]*[^#]/ { print $1; exit }')
umount $TARGETPATH
vgchange -an
cryptsetup close temp_name
# This proper name is required for `update-initramfs` to properly update things
cryptsetup open $CRYPTPATH $CRYPTNAME
wait
vgchange -ay
ROOTPATH=$(ls /dev/mapper/* | grep root)
mount $ROOTPATH $TARGETPATH
mount $BOOTPATH $TARGETPATH/boot
mount $EFIPATH $TARGETPATH/boot/efi
mount -t proc proc $TARGETPATH/proc
mount -o bind /dev $TARGETPATH/dev
# Have also seen people mounting dev/pts and run and sys, they don't appear to be necessary

chroot $TARGETPATH update-initramfs -c -k all
echo "Completed crypt-fix, try rebooting and you should get prompted for your passphrase after grub"

Comments

[JohnW]

It depends on a lot of things from an hardware failure (bit flip included) to an upgrade failure. So basically you have to try everything from a full grub reinstall to trying to boot with the previous kernel.

[knocte]

how to do full grub reinstall? wrt kernel, I already said I cannot boot with a previous kernel because I only had 1 installed

[JohnW]

When did Ubuntu stop to provide previous kernels? There was a reason for this: basically to avoid the situation you are in and to provide a meaningful feedback in case of a bug. As it is now, helping you to solve this problem would require hours of my own time: I won't do it. So my (quick) advice would be to 1. reinstall grub (grub-install) 2. update its configuration (grub-mkconfig) 3. if not solved, regenerate the initramfs (mkinitcpio or equivalent) 4. if not solved, chroot from liveCD and reinstall previous kernel. If nothing worked, reinstall. (note the cmds are not complete)