ZFS on Linux/Ubuntu: Help importing a zpool after Ubuntu upgrade from 13.04 to 13.10, device IDs have changed

Question

I have updated Ubuntu from version 13.04 to version 13.10, only to discover that the SATA disks IDs have changed, and my ZFS pool now fails.

On new Ubuntu 13.10 the disk IDs are now ata-* instead of scsi-SATA_*.

This is the pool status after the update:

  pool: nestpool
 state: UNAVAIL
status: One or more devices could not be used because the label is missing 
    or invalid.  There are insufficient replicas for the pool to continue
    functioning.
action: Destroy and re-create the pool from
    a backup source.
   see: http://zfsonlinux.org/msg/ZFS-8000-5E
  scan: none requested
config:

    NAME                                                STATE     READ WRITE CKSUM
    nestpool                                            UNAVAIL      0     0     0  insufficient replicas
      raidz2-0                                          UNAVAIL      0     0     0  insufficient replicas
        scsi-SATA_WDC_WD4000F9YZ-_WD-WCC1F0046946       UNAVAIL      0     0     0
        scsi-SATA_WDC_WD4000F9YZ-_WD-WCC4A0026423       UNAVAIL      0     0     0
        scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0011145       UNAVAIL      0     0     0
        scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0049294       UNAVAIL      0     0     0
        scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0051143       UNAVAIL      0     0     0
        scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0051756       UNAVAIL      0     0     0
        scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0056625       UNAVAIL      0     0     0
        scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0200560       UNAVAIL      0     0     0
    logs
      mirror-1                                          UNAVAIL      0     0     0  insufficient replicas
        scsi-SATA_Samsung_SSD_840S1ATNEAD707062H-part2  UNAVAIL      0     0     0
        scsi-SATA_Samsung_SSD_840S1ATNEAD707066K-part3  UNAVAIL      0     0     0

After a very long research on the Internet I started following procedure:

First exported the pool using: zpool export nestpool

Then tried to import back the pool using: zpool import -m -f -d /dev/disk/by-id nestpool

But the import fails with the message: cannot import 'nestpool': one or more devices is currently unavailable

This is the current output of zpool import

   pool: nestpool
     id: 3947768928242827823
  state: DEGRADED
 status: One or more devices contains corrupted data.
 action: The pool can be imported despite missing or damaged devices.  The
    fault tolerance of the pool may be compromised if imported.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
 config:

    nestpool                                                DEGRADED
      raidz2-0                                              ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WCC1F0046946          ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WCC4A0026423          ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0011145          ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0049294          ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0051143          ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0051756          ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0056625          ONLINE
        ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0200560          ONLINE
    cache
      ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707062H-part1
    logs
      mirror-1                                              UNAVAIL  insufficient replicas
        ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707062H      UNAVAIL  corrupted data
        ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707066K      UNAVAIL

Log partitions were not correctly identified, they should be ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707062H-part2 and ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707066K-part3. And the argument -m during import is not helping.

I cannot find a way to tell zpool import to use a different path/id for the log devices. Any help and idea for fixing this problem will be much appreciated. What else can I do to recover this pool?

Solution

Please try zpool list. This will show if the pool is even available to the system.

Try to import. Maybe a zpool import -f nestpool

Perhaps try to remove the unavailable log devices via:

zpool remove nestpool mirror-1

From now on, use whole devices for L2ARC and ZIL...

Edit:

Your easiest fix is to temporarily create the symbolic links you need in /dev/disk/by-id in order to import the pool. An example from a RHEL system running the current ZFS.

[root@Davalan /dev/disk/by-id]# ll
total 0
lrwxrwxrwx 1 root root  9 Oct 27 05:29 ata-STEC_M8IOPS-50_STM000136649 -> ../../sdc
lrwxrwxrwx 1 root root 10 Oct 27 05:29 ata-STEC_M8IOPS-50_STM000136649-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Oct 27 05:29 ata-STEC_M8IOPS-50_STM000136649-part9 -> ../../sdc9
lrwxrwxrwx 1 root root  9 Oct 27 05:29 scsi-35000c5003af99fa7 -> ../../sdd
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000c5003af99fa7-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000c5003af99fa7-part9 -> ../../sdd9
lrwxrwxrwx 1 root root  9 Oct 27 05:29 scsi-35000cca0153ec2d0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000cca0153ec2d0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000cca0153ec2d0-part9 -> ../../sdb9
lrwxrwxrwx 1 root root  9 Oct 27 05:29 scsi-35000cca01540e298 -> ../../sdf
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000cca01540e298-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000cca01540e298-part9 -> ../../sdf9
lrwxrwxrwx 1 root root  9 Oct 27 05:29 scsi-35000cca01540e340 -> ../../sde
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000cca01540e340-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-35000cca01540e340-part9 -> ../../sde9
lrwxrwxrwx 1 root root  9 Oct 27 05:29 scsi-SATA_STEC_M8IOPS-50_STM000136649 -> ../../sdc
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-SATA_STEC_M8IOPS-50_STM000136649-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Oct 27 05:29 scsi-SATA_STEC_M8IOPS-50_STM000136649-part9 -> ../../sdc9

Comments

[Manolo]

Once I exported the pool, it was automatically removed from the system, so it is no longer available to the system, actually this is why I need help. I think this is the normal behaviour. I am sorry again if it is not, in this case I also forgot to mention it. And so all these commands do not apply.

[Manolo]

I wish there was a way to apply similar commands to exported pools.

[Manolo]

smart idea the one with the links, gonna try now...

[Manolo]

Cooooooool, it worked. It first changed the devices again to something else, like this: mirror-1 UNAVAIL insufficient replicas wwn-0x50025385503e8531 UNAVAIL corrupted data wwn-0x50025385503e8535 UNAVAIL but then I applied the linking trick to these new names... and it finally worked!

[ewwhite]

Now, remove the ZIL devices and re-add them with the system-generated device names.

[Manolo]

And I will for sure no longer use partitions for anything related to ZFS! Lesson learned!