Recovering overwritten LVM metadata

Question

Yesterday I was trying using the Debian install in rescue mode in order to fix my Grub (The menu-item for Debian was suddenly gone).

Now I wasn't able to fix it and didn't have a lot of time, so I thought I'd just reinstall Debian (I know, windows style :P). Problem was that I had installed Debian using full disk encryption, thus within a LVM partition. For some reason the installer didn't recognize the volume groups, though I could mount them in rescue mode. I tried to set "use as lvm" and then select option "use current layout", but no volume groups were present anymore. Going back to rescue mode, executing a shell, I couldn't mount them anymore, as if they were gone.

So somewhere along the way the metadata for the volume groups must have been overwritten. I did some googling, but hardly found anything on the specific subject of overwrittten metadata. I've tried running testdisk, but it seems that's only for recovering partitions, not volume groups.

Last I could find was the output of pvck:

root@ubuntu:/home/ubuntu# pvck -d -v /dev/sda5
    Scanning /dev/sda5
  Found label on /dev/sda5, sector 1, type=LVM2 001
  Found text metadata area: offset=4096, size=192512
    Found LVM2 metadata record at offset=194560, size=2048, offset2=0 size2=0
    Found LVM2 metadata record at offset=128512, size=66048, offset2=0 size2=0
    Found LVM2 metadata record at offset=116224, size=12288, offset2=0 size2=0
    Found LVM2 metadata record at offset=69120, size=47104, offset2=0 size2=0
    Found LVM2 metadata record at offset=68096, size=1024, offset2=0 size2=0
    Found LVM2 metadata record at offset=65024, size=3072, offset2=0 size2=0
    Found LVM2 metadata record at offset=53248, size=11776, offset2=0 size2=0
    Found LVM2 metadata record at offset=52736, size=512, offset2=0 size2=0
    Found LVM2 metadata record at offset=51712, size=1024, offset2=0 size2=0

Now that does look like it found something right? But unfortunately I don't know enough about LVM's, so I have no idea how to use this information. Anyone who is able to guide me further?

Extra information:

output of pvscan:

root@ubuntu:/home/ubuntu# pvscan
  PV /dev/sda5                      lvm2 [465.52 GiB]
  Total: 1 [465.52 GiB] / in use: 0 [0   ] / in no VG: 1 [465.52 GiB]

I am testing on live Ubuntu 10.04 LTS 32-bit live-cd and I was trying to reinstall Debian Squeeze 64-bit. I am using a Acer Aspire laptop with intel i7 quad-core processor, 2x 1TB internal hard drives, ati Radeon HD 5600 Series videocard. (which is probably more information than needed :P).

Solution 1

Gave up and formatted the drive containing the lvm as I was really in need of more space.

Solution 2

If you have system backups that were made after the last change made to the LVM, most LVM commands backup the LVM metadata to /etc/, and there is a way to recover that data to the disk.

I would recommend you to find a new hard drive and use dd to copy all the data on the drive you're trying to recover and work on the copy, because any mistake in the recovery might reduce the chances of recovering to 0.

Explanation how to recover LVM metadata from backup

Don't keep your hopes up, the full disk encryption keeps it's keys in a small area in the beginning of the disk, and when data is deleted, it's too easy to erase the keys.

Solution 3

You've lost all your data. May as well get that out of the way, so you can move on to other things.

The Debian installer will not work with existing encrypted installations, and just happily overwrites them with fresh LVM metadata even if told to ignore them. It's not the LVM metadata you're missing, but the LUKS metadata that used to be in the PV.

This also applies to Ubuntu systems, which use the same code.

I got burned by this not long after squeeze came out when I decided I wanted to set up a dual boot with Fedora and Debian, and consequently lost the entire contents of my hard drive (including many things not backed up).

The take-away here is: Make backups and make sure they work.

Comments

[Admin]

I don't know lvm well enough to figure out what was overwritten, but there's a good chance that the key to the volume was part of the overwritten data. In which case you aren't going to recover anything. I recommend going straight to restoring from backups.

[Admin]

That's the problem there are some projects I was working on, which I just started and have no back-up for :(.

[Joren Van Severen]

I never make full system backups, only for home-partition. I have been to that site earlier, nothing useful, I don't even get an error message :(.

[jasonwryan]

Please don't just post a link: include some relevant details so that if the site goes down the information is still accessible...

[Razzlero]

So you're basically locked out of your system.

[Joren Van Severen]

tried that already, didn't work