How to recover resize2fs failure

debian vmware-esxi drive-failure

5,648

Solution 1

First, resize2fs may take several hours or even more than a day to run, depending on the size of your partition.

If you interrupted it while it was running, that's a very good recipe for losing all your data. All of the filesystem data structures are probably in an inconsistent state.

Now, your drive has errors at the very beginning of the disk, where the partition table is stored.

At this point, what you should do is replace the disk and restore your backups. Since you don't have backups, you should update your CV. If this server was very important, the company might go out of business.

To avoid situations like this in future, always ensure that you have backups and that your backups are fully functional, that you can successfully restore with them.

So, aside from updating your CV and making backups, this is what you need to do today:

You can attempt to use dd_rescue to copy the disk, skipping any errors, to another disk, and then work on rescuing data from the copy.

Don't expect to get too much out of this, though.
If you can't rescue anything from the disk, send it to a professional data recovery company.

Solution 2

I know this is years later and not likely helpful to the OP, but I wanted to add another perspective on this for others coming across this post.

I've had resize2fs interrupted more than once (i.e. power failures), and after a quick fsck have not had any issues or data loss. It certainly sounds plausible to lose large amounts of data this way or even corrupt the entire partition since this is affecting the filesystem itself, I've just never personally witnessed it before.

However, the errors the OP posted combined with failed attempts to run dd_rescue (which doesn't care about corruption at the filesystem level) points to a hardware failure, (or in the case of a VM, a problem with the container file or perhaps the underlying hardware). While coincidental, this sounds independent of the failed resize2fs attempt.

I would first make sure the underlying hardware is good. If all checks out there, make a copy of the VM image, then rebuild the copy (the specific method for that will depend on the image format). Finally, boot the copy, running fsck as well, and see if that resolves the issue.

5,648

tdbui22

Updated on September 18, 2022

Comments

tdbui22 almost 2 years

I was resizing my hard drive last night and was not successful.

My system and drives are local on an ESXi 5.1 vm.

I'm running Debian 6 x64 and have a 2TB mount that I was resizing. It was about 1.8T that I was resizing to the full 2TB.

I ran e2fsck on /dev/sdc1 and it had no issues. I than ran resize2fs and let it run for a few hours. After checking ps, it showed the state in sleep (D+) with no activity. I couldn't kill the process so I rebooted and now I cannot do anything with the drive.

Now I'm missing the /dev/sdc1 drive and I cannot do anything that will read that drive; fdisk, cfdisk, e2fsck, resize2fs all freeze attempting to do anything related to this drive.

What can I do to recover this drive? I unfortunately do not have a backup of this drive. I just want to be able to mount the drive to recover my data or at least read the data on the drive somehow.

dmesg output

[  722.816205] sd 2:0:2:0: timing out command, waited 180s
[  722.816329] sd 2:0:2:0: [sdc] Unhandled error code
[  722.816336] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  722.816343] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[  722.816352] end_request: I/O error, dev sdc, sector 0
[  722.816443] Buffer I/O error on device sdc, logical block 0
[  722.816534] Buffer I/O error on device sdc, logical block 1
[  722.816619] Buffer I/O error on device sdc, logical block 2
[  722.816703] Buffer I/O error on device sdc, logical block 3
[  903.215677] sd 2:0:2:0: timing out command, waited 180s
[  903.215802] sd 2:0:2:0: [sdc] Unhandled error code
[  903.215805] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  903.215808] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[  903.215817] end_request: I/O error, dev sdc, sector 0
[  903.215943] Buffer I/O error on device sdc, logical block 0
[  903.216037] Buffer I/O error on device sdc, logical block 1
[  903.216124] Buffer I/O error on device sdc, logical block 2
[  903.216211] Buffer I/O error on device sdc, logical block 3
[  903.216337] ldm_validate_partition_table(): Disk read failed.
[ 1083.615097] sd 2:0:2:0: timing out command, waited 180s
[ 1083.615216] sd 2:0:2:0: [sdc] Unhandled error code
[ 1083.615218] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1083.615222] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1083.615231] end_request: I/O error, dev sdc, sector 0
[ 1083.615316] Buffer I/O error on device sdc, logical block 0
[ 1083.615411] Buffer I/O error on device sdc, logical block 1
[ 1083.615496] Buffer I/O error on device sdc, logical block 2
[ 1083.615580] Buffer I/O error on device sdc, logical block 3
[ 1264.014510] sd 2:0:2:0: timing out command, waited 180s
[ 1264.014667] sd 2:0:2:0: [sdc] Unhandled error code
[ 1264.014669] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1264.014672] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1264.014680] end_request: I/O error, dev sdc, sector 0
[ 1264.014766] Buffer I/O error on device sdc, logical block 0
[ 1264.014855] Buffer I/O error on device sdc, logical block 1
[ 1264.014939] Buffer I/O error on device sdc, logical block 2
[ 1264.015022] Buffer I/O error on device sdc, logical block 3
[ 1444.413968] sd 2:0:2:0: timing out command, waited 180s
[ 1444.414098] sd 2:0:2:0: [sdc] Unhandled error code
[ 1444.414105] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1444.414113] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1444.414124] end_request: I/O error, dev sdc, sector 0
[ 1444.414246] Buffer I/O error on device sdc, logical block 0
[ 1444.414363] Buffer I/O error on device sdc, logical block 1
[ 1444.414450] Buffer I/O error on device sdc, logical block 2
[ 1444.414536] Buffer I/O error on device sdc, logical block 3
[ 1624.813488] sd 2:0:2:0: timing out command, waited 180s
[ 1624.813596] sd 2:0:2:0: [sdc] Unhandled error code
[ 1624.813603] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1624.813611] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1624.813620] end_request: I/O error, dev sdc, sector 0
[ 1624.813699] Buffer I/O error on device sdc, logical block 0
[ 1624.813786] Buffer I/O error on device sdc, logical block 1
[ 1624.813870] Buffer I/O error on device sdc, logical block 2
[ 1624.813953] Buffer I/O error on device sdc, logical block 3
[ 1624.814071] Dev sdc: unable to read RDB block 0
[ 1805.212836] sd 2:0:2:0: timing out command, waited 180s
[ 1805.212951] sd 2:0:2:0: [sdc] Unhandled error code
[ 1805.212953] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1805.212956] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1805.212963] end_request: I/O error, dev sdc, sector 0
[ 1805.213052] Buffer I/O error on device sdc, logical block 0
[ 1805.213143] Buffer I/O error on device sdc, logical block 1
[ 1805.213230] Buffer I/O error on device sdc, logical block 2
[ 1805.213317] Buffer I/O error on device sdc, logical block 3
[ 1985.612363] sd 2:0:2:0: timing out command, waited 180s
[ 1985.612556] sd 2:0:2:0: [sdc] Unhandled error code
[ 1985.612558] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1985.612561] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1985.612568] end_request: I/O error, dev sdc, sector 0
[ 1985.612691] Buffer I/O error on device sdc, logical block 0
[ 1985.612785] Buffer I/O error on device sdc, logical block 1
[ 1985.612878] Buffer I/O error on device sdc, logical block 2
[ 1985.612966] Buffer I/O error on device sdc, logical block 3
[ 2166.011692] sd 2:0:2:0: timing out command, waited 180s
[ 2166.011806] sd 2:0:2:0: [sdc] Unhandled error code
[ 2166.011808] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2166.011811] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2166.011818] end_request: I/O error, dev sdc, sector 24
[ 2166.011899] Buffer I/O error on device sdc, logical block 12
[ 2166.011998] Buffer I/O error on device sdc, logical block 13
[ 2166.012083] Buffer I/O error on device sdc, logical block 14
[ 2166.012168] Buffer I/O error on device sdc, logical block 15
[ 2346.411194] sd 2:0:2:0: timing out command, waited 180s
[ 2346.411377] sd 2:0:2:0: [sdc] Unhandled error code
[ 2346.411384] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2346.411392] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2346.411403] end_request: I/O error, dev sdc, sector 24
[ 2346.411505] Buffer I/O error on device sdc, logical block 12
[ 2346.411628] Buffer I/O error on device sdc, logical block 13
[ 2346.411730] Buffer I/O error on device sdc, logical block 14
[ 2346.411829] Buffer I/O error on device sdc, logical block 15
[ 2526.810657] sd 2:0:2:0: timing out command, waited 180s
[ 2526.810824] sd 2:0:2:0: [sdc] Unhandled error code
[ 2526.810827] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2526.810831] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2526.810841] end_request: I/O error, dev sdc, sector 0
[ 2526.810978] Buffer I/O error on device sdc, logical block 0
[ 2526.811096] Buffer I/O error on device sdc, logical block 1
[ 2526.811184] Buffer I/O error on device sdc, logical block 2
[ 2526.811271] Buffer I/O error on device sdc, logical block 3
[ 2526.811391]  unable to read partition table
[ 2526.811899] sd 2:0:2:0: [sdc] Cache data unavailable
[ 2526.811901] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[ 2526.811998] sd 2:0:2:0: [sdc] Attached SCSI disk
[ 2706.830303] VMware vmxnet3 virtual NIC driver - version 1.0.5.0-k-NAPI
[ 2706.833247] VMware PVSCSI driver - version 1.0.1.0-k
[ 2706.836002] VMware vmxnet virtual NIC driver
[ 2706.845581] PM: Starting manual resume from disk
[ 2706.845584] PM: Resume from partition 8:5
[ 2706.845586] PM: Checking hibernation image.
[ 2706.845921] PM: Error -22 checking image file
[ 2706.845923] PM: Resume from disk failed.
[ 2706.858105] kjournald starting.  Commit interval 5 seconds
[ 2706.858117] EXT3-fs: mounted filesystem with ordered data mode.
[ 2707.043045] udev[390]: starting version 164
[ 2707.070265] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[ 2707.070276] ACPI: Power Button [PWRF]
[ 2707.075278] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 2707.077419] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 2707.093012] [415]: VMCI: shared components initialized.
[ 2707.093068] Probing for vmci/PCI.
[ 2707.093106]   alloc irq_desc for 16 on node -1
[ 2707.093107]   alloc kstat_irqs on node -1
[ 2707.093117] vmci 0000:00:07.7: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 2707.093196] Found vmci/PCI at 0x1080, irq 16.
[ 2707.093226] VMCI: using capabilities 0xc.
[ 2707.093327] [415]: VMCI: Host capability check passed.
[ 2707.093471]   alloc irq_desc for 56 on node -1
[ 2707.093472]   alloc kstat_irqs on node -1
[ 2707.093524] vmci 0000:00:07.7: irq 56 for MSI/MSI-X
[ 2707.093525]   alloc irq_desc for 57 on node -1
[ 2707.093526]   alloc kstat_irqs on node -1
[ 2707.093546] vmci 0000:00:07.7: irq 57 for MSI/MSI-X
[ 2707.093717] Registered vmci device.
[ 2707.093760] [415]: VMCI: Using guest personality
[ 2707.093762] [415]: VMCI: host components initialized.
[ 2707.093785] [415]: VMCI: Module registered (name=vmci,major=10,minor=59).
[ 2707.093786] [415]: VMCI: Using host personality
[ 2707.093787] [415]: VMCI: Module (name=vmci) is initialized
[ 2707.102777] processor LNXCPU:00: registered as cooling_device0
[ 2707.103353] processor LNXCPU:01: registered as cooling_device1
[ 2707.103522] piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!
[ 2707.104218] processor LNXCPU:02: registered as cooling_device2
[ 2707.105603] processor LNXCPU:03: registered as cooling_device3
[ 2707.107367] input: PC Speaker as /devices/platform/pcspkr/input/input3
[ 2707.117231] ACPI: AC Adapter [ACAD] (on-line)
[ 2707.148586] parport_pc 00:09: reported by Plug and Play ACPI
[ 2707.148765] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[ 2707.193924] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[ 2707.199564] Error: Driver 'pcspkr' is already registered, aborting...
[ 2707.209963] sd 2:0:2:0: timing out command, waited 180s
[ 2707.210018] sd 2:0:2:0: [sdc] Unhandled error code
[ 2707.210020] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2707.210022] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2707.210029] end_request: I/O error, dev sdc, sector 0
[ 2707.210077] Buffer I/O error on device sdc, logical block 0
[ 2707.210129] Buffer I/O error on device sdc, logical block 1
[ 2707.210179] Buffer I/O error on device sdc, logical block 2
[ 2707.210230] Buffer I/O error on device sdc, logical block 3
[ 3787.606904] sd 2:0:2:0: [sdc] Unhandled error code
[ 3787.606907] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 3787.606910] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 3787.606921] end_request: I/O error, dev sdc, sector 0
[ 3787.607032] Buffer I/O error on device sdc, logical block 0
[ 3787.607145] Buffer I/O error on device sdc, logical block 1
[ 3787.607256] Buffer I/O error on device sdc, logical block 2
[ 3787.607367] Buffer I/O error on device sdc, logical block 3
[ 4868.003267] sd 2:0:2:0: timing out command, waited 1080s
[ 4868.003392] sd 2:0:2:0: [sdc] Unhandled error code
[ 4868.003394] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 4868.003397] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 4868.003407] end_request: I/O error, dev sdc, sector 0
[ 4868.003495] Buffer I/O error on device sdc, logical block 0
[ 4868.003595] Buffer I/O error on device sdc, logical block 1
[ 4868.003685] Buffer I/O error on device sdc, logical block 2
[ 4868.003774] Buffer I/O error on device sdc, logical block 3
[ 5948.399889] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400005] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400012] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400019] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 5948.400028] end_request: I/O error, dev sdc, sector 0
[ 5948.400110] Buffer I/O error on device sdc, logical block 0
[ 5948.400209] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400294] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400295] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400298] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 5948.400304] end_request: I/O error, dev sdc, sector 2
[ 5948.400384] Buffer I/O error on device sdc, logical block 1
[ 5948.400473] Buffer I/O error on device sdc, logical block 2
[ 5948.400562] Buffer I/O error on device sdc, logical block 3
[ 7028.796518] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796633] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796640] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796648] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 7028.796657] end_request: I/O error, dev sdc, sector 0
[ 7028.796746] Buffer I/O error on device sdc, logical block 0
[ 7028.796864] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796948] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796949] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796952] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 7028.796958] end_request: I/O error, dev sdc, sector 2
[ 7028.797037] Buffer I/O error on device sdc, logical block 1
[ 7028.797122] Buffer I/O error on device sdc, logical block 2
[ 7028.797207] Buffer I/O error on device sdc, logical block 3

tdbui22 about 11 years

I still have the /dev/sdc, but no longer have /dev/sdc1.
Sergey Vlasov about 11 years

Also make sure that /dev/sdc is really the drive you think it is — sdX device assignments can change, e.g., if some device was not detected, or in some cases they may change every boot if you have multiple controllers detected in parallel.
tdbui22 about 11 years

@SergeyVlasov I've added the dmesg output to my question. I also forgot to mention that this is a esxi vm drive. It is definitely the correct drive as well.
Sergey Vlasov about 11 years

Do you have admin access to the ESXi server, and on what kind of storage is that virtual drive located? Also, was the VM just rebooted, or stopped completely and restarted?
tdbui22 about 11 years

@SergeyVlasov I do have admin access to the server. All storage is on a local sata drive. The vm was rebooted.
Sergey Vlasov about 11 years

Then you should look for any storage-related errors or warnings on the ESXi side (dying physical drive, full datastore, VMFS corruption, …); you probably will not get useful information from inside a VM. I don't have experience working with ESXi 5.1, so cannot give exact recommendation on where to look.
tdbui22 about 11 years

@SergeyVlasov the problem was with ESXi. The drive was marked as inactive and I couldn't do anything with it. I rebooted it, unmounted it, rebooted, remounted and it worked again. Thanks!
voretaq7 about 11 years

I unfortunately do not have a backup of this drive. <-- And after this little scare you're not going to make that mistake again, are you? :-)

tdbui22 about 11 years

Unfortunately I do not have backup and in serious crisis at the moment. I just tried dd_rescue and it just freezes. Checking ps and it is in sleep state. Everything that touches this drive goes to sleep state.