Can't remove LVM thinpool - unable to deactivate tdata, tmeta

Question

I somehow messed up some thinly provisioned lvm volumes on ubuntu 14.04 and now I want to start fresh by deleting the thinpool with all its volumes and data inside. Unfortunatelly this failes and I can't find a solution.

The logical volumes look like this:

user@server1:~$ sudo lvs
  dm_report_object: report function failed for field data_percent
  LV             VG   Attr      LSize   Pool       Origin Data%  Move Log Copy%  Convert
  project2       vg0  Vwi-i-tz-  22.00g mythinpool       
  project1       vg0  Vwi---tz-  20.00g mythinpool                                      
  project3       vg0  Vwi---tz-  21.00g mythinpool                                      
  home           vg0  -wi-ao--- 140.00g                                                 
  mythinpool     vg0  twi-i-tz-  78.82g                    52.15                        
  root           vg0  -wi-ao---  10.00g                                                 
  swap           vg0  -wi-ao---   4.00g                                                 
  tmp            vg0  -wi-ao---   5.00g           

Now I want to remove the thinpool with the three lvms inside:

sudo lvremove /dev/vg0/mythinpool

Removing pool mythinpool will also remove 3 thin volume(s). OK? [y/n]: y
Do you really want to remove and DISCARD logical volume project1? [y/n]: y
  device-mapper: message ioctl on  failed: Invalid argument
  Unable to deactivate open vg0-mythinpool_tdata (252:5)
  Unable to deactivate open vg0-mythinpool_tmeta (252:4)
  Failed to deactivate vg0-mythinpool-tpool
  Failed to resume mythinpool.
  Failed to update thin pool mythinpool.

I don't care about the data inside mythinpool, but the rest of the volume group vg0 MUST stay intact. How could I solve this problem? Thank you for any help on this.

EDIT 1: After following the answer from shodanshok I was able to remove one LVM-Image by booting into CentOS7, but unfortunatelly the other two volumes including thin pool return another error message - transaction_id mismatch:

There is also no space available for lvconvert --repair

Solution 1

I finaly solved it very with some simple steps as described here: remove corrupt LVM thin pool

vgcfgbackup -f  vg.bak   vgname

//edit vg.bak and remove all thinp related volumes

vgcfgrestore -f vg.bak  vgname

After following the advice from @shodanshok to boot into Live CentOS7 sticked to the server on an USB stick, I was able to issue the described commands and to eventually get rid of the corrupt thin pool without damaging the root file system that resides inside the same volume group.

Thank you everybody for your helpful advice that led eventually to the solution.

Solution 2

Something is keeping your thin volumes opened. Please do the following:

• check if the volumes are mounted. If so, unmount them
• if you can not unmount them, maybe some files are still opened by a running process. You can use lsof | grep mountpoint to find the offending processes. Kill them and try to unmount the filesystems
• if this fails, reboot the machine and try to unmount the filesystems related to the thin volumes
• if you still can't unmount the volumes, try to reboot into single user mode, unmount them and delete the volumes
• if all else fails, use a live CD/USB image to boot your PC and remove the offending volumes.

EDIT:

As you can't use a live image and your rescue system has no thin-volume support, we can try an alternate route. Basically, we will set the "skip activation flag" on your thin volumes/pool and reboot the machine. Follow these steps:

• set activation skip: lvchange -ky vg0/project1 ; lvchange -ky vg0/project2 ; lvchange -ky vg0/project3 ; lvchange -ky vg0/mythinpool
• reboot the machine and the volumes should be inactive
• you can now remove them with lvremove

However, if any of these volumes are needed for machine boot you will end with an unbootable machine. Be sure to have a "plan B" to restore the machine via a recovery console or such.

EDIT 2

If your system does not support the -k flag, you can try using lvchange -aay volumename and reboot. This will set the volume for autoactivation, which only works for volumes specified on /etc/lvm.conf

Comments

[merlin]

There was nothing mounted, so I now rebooted into rescue mode in order to remove them. During boot the message came up "Setting up lvm Volume Groups failed". I then tried to remove, but this failed as well: root@rescue ~ # lvremove /dev/vg0/project1 WARNING: Unrecognised segment type thin-pool WARNING: Unrecognised segment type thin Cannot change VG vg0 with unknown segments in it! Skipping volume group vg0

[shodanshok]

It happear that rescue mode does not load the required kernel modules. The easy solution is to simply boot off the Ubuntu Live Image, open a root console and delete the volumes.

[merlin]

That is unfortunatelly not so easy as the server is inside a remote data center. I can boot a rescue system easy but other then that it will get complicated and expensive. Is there a another solution to the problem?

[shodanshok]

From here it seems that the error you described depends on LVM not been compiled with thin volumes support. Unfortunately, it seems you can not remove volumes within your rescue environment. I'll edit my answer to post another possible solution.

[merlin]

That sounds like a plan! Unfortunatelly I do get: lvchange -ky vg0/project1 lvchange: invalid option -- 'k'. I also tried: lvchange --setactivationskip y vg0/project1 which is also unrecognized. I am on ubunut 14.04.03

[merlin]

In the meantime I managed to get an ubuntu 14.04 image mounted via an USB-Stick. Would this help? Looks like you ment another image as this seems to be a regular installation iso. I can boot this one: releases.ubuntu.com/14.04/ubuntu-14.04.3-server-amd64.iso from USB.

[shodanshok]

Ubuntu installation isos can be used to live-boot, too. Anyway, if you can boot off a USB drive, I suggest you to use a CentOS 7 live image, which has excellent out of the box support for thin volumes.

[shodanshok]

I've edited the answer to show another possibility yet.

[merlin]

Thank you. I wish it would work, but I do get this message: sudo lvchange -aay vg0/project1 device-mapper: message ioctl on failed: Invalid argument Unable to deactivate open vg0-mythinpool_tdata (252:5) Unable to deactivate open vg0-mythinpool_tmeta (252:4) Failed to deactivate vg0-mythinpool-tpool