Missing root VG, dropped to an initramfs shell

Question

We are facing an issue while pulling out data from Listen 360 API. We are using following Code:

class Program
{
    private const string URL =      "https://9833bf430be5620b7fa941ee5c3c7e2add61a86e:[email protected]/organizations/594214508264690689/reviews.xml";


    static void Main(string[] args)
    {
        HttpClient client = new HttpClient();
        client.BaseAddress = new Uri(URL);         

        // List data response.
        HttpResponseMessage response = client.GetAsync(URL).Result;  // Blocking call!
        if (response.IsSuccessStatusCode)
        {
            // Parse the response body. Blocking!              
        }
        else
        {
            Console.WriteLine("{0} ({1})", (int)response.StatusCode, response.ReasonPhrase);
        }
    }
}

When the URL is directly hit on browser the XML response is recieved. But when we are trying to access it programatically we get "Unauthorized 401" response

Solution 1

Make sure your "/boot/grub/grub.cfg" lists the correct VG and LV. In my case, the VG is named "ssd_vg" and the LV is named "root_lv", so the grub.cfg file needs to reference "/dev/mapper/ssd_vg-root_lv" (note use of slashes and dashes!).

Solution 2

On Debian-based systems if your partition is LUKS encrypted before running update-initramfs command make sure that cryptsetup-initramfs package is installed. If not, install it first:

apt-get install -y cryptsetup-initramfs

Solution 3

in initramfs type

 #/sbin/lvm vgchange -a y
 # vgchange -a y
 #exit

if you unable to solve the issue type this

  #ls /dev/mapper

you found root (like /dev/vgname/lvname) #reboot then select a kernel on a screen and select e to edit and paste a root value (like /dev/vgname/lvname) at /boot/vmlinuxxxxx root=uuid/dev/xxx. then ctrl+x to boot

If problem repeats when reboot the server

go to

      #/boot/grub/grub.cfg

at /vm/vmlinuz root=write your full lv path

if problem not fixed try fix initramfs as below

1.Go to server terminal

  # sudo rmmod floppy

    #echo "blacklist floppy" | sudo tee /etc/modprobe.d/blacklis-floppy.conf
     #dpkg-reconfigure initramfs-tools
     #update-initramfs -u
     #update-grub
     #reboot

2. Execute this

       #/etc/initramfs-tools/scripts/local-top/forcelvm

with the following contents:

       #!/bin/sh
       PREREQ=""
     prereqs()
        {
       echo "$PREREQ"
     }
     case $1 in
     prereqs)
     prereqs
    exit 0
      ;;
    esac
      . /scripts/functions
       lvm vgchange -ay

Then do

      # chmod +x `/etc/initramfs-tools/scripts/local-top/forcelvm`
       # update-initramfs -u -k all

3. take backup of lvm2

    #apt-get install lvm2
    #cp /usr/share/initramfs-tools/scripts/local-top/lvm2  /tmp
   

   Edit lvm2

       #vi /usr/share/initramfs-tools/scripts/local-top/lvm2
   

   write Between modprobe -q dm-mod and activate_vg "$ROOT" add this line to initialize your lvm:

     lvm vgchange -ayactivate_vg "$ROOT" if you not find this line
   

write below two line at the end of the file above exit 0

       #lvm vgchange -ayactivate_vg "$ROOT"
       #activate_vg="$ROOT"

save the file then

        #update-initramfs -u
        #update-grub
        #reboot

Comments

[Admin]

usually if it's saying a Volume Group isn't accessible, it's because there are too many physical volumes missing. The root of the issue in that case would be to see why it's not able to find all the pv's. Not sure what ubuntu puts into their initramfs but you might try checking the output of both fdisk -l and pvs to see if any of your volumes are missing. Removing the ssd isn't likely to fix the problem and is probably the wrong thing to do.

[Admin]

They're not going to remove LVM support mid-stream on any distro, so you can forget the stuff where people are saying the kernel might not support it. That could cause this, but it's not what's hitting you. Also, this shouldn't stop you from booting from DVD, though. You might have to check your boot order for that.

[Admin]

Thanks, for both your comments. I can type 'lvm' in the initramfs shell, and it says that my OS PV+VG+LV are active and they seem okay. I'll try 'fdisk -l' in the morning.

[Admin]

the above output is saying the device file for your LV doesn't exist at all which usually only happens if the logical volume isn't active. You may check the output of lvm lvs to ensure the logical volume shows as active and not just that it's there (it's possible that it's just not able to find enough extents for the logical volume itself). lvm pvs might also tell you which PV's it actually did find and you can figure out which one is missing by process of elimination.

[Admin]

I've added more output, looks like all volumes are in similar state (either all good or all bad)?

[Admin]

Try booting with an older kernel in the boot menu or else try with the kernel option root=/dev/mapper/ssd_vg-root_lv . I suspect somehow the new initramfs was created with a default volume group of 'mint-' (?) instead of your ssd_vg.

[Admin]

@nkms, I have two kernels available: 3.13.0-46-generic (which is the default) and, under "previous linux versions", 3.13.0-24-generic. Using either of these yields no difference as far as I can see. :( But yes, I think I did rename the VG's at some point (far earlier than the latest reboot, though). Can I update the naming in the initramfs?

[Admin]

Provided that you can boot giving the kernel option root=/dev/mapper/ssd_vg-root_lv in the grub menu then you can update iniramfs with update-initramfs. However, if you have renamed the volume than you need to modify /etc/fstab also. Might be easier to rename back the volume ether from initramfs or a live cd (because the name mint--vg-root does not make it clear which is the volume group name and the logical volume name I can't give the exact commands. The slash - is a separator but there's three of them in that name).

[Admin]

Should I try using "boot-repair" as suggested in other posts? Or do I then risk damaging my lvm's further?

[Admin]

I did a daring thing, it seems to have worked.

[Admin]

@KlaymenDK - Congratulations! Consider answering your own question and accepting it as an answer as opposed to marking topic as solved - this will be both clearer and will allow others to improve your solution.

[Admin]

@MatthewRock, patience. :-) I did make my own answer yesterday, but could not select it as the chosen solution until today.

[Bratchley]

tbh it sounds like you renamed the VG without updating grub. It's important to remember to regenerate initramfs and update grub if you rename the VG. Installing a new kernel probably generated a new initramfs for you by itself though. Glad you found the solution, though.