Why is fsck saying corrupt superblock or partition table and how to fix?

Question

I made an image (dd) of a drive and I am trying to run a filesystem check on it: Filesystem Type: ext3

Here is the Original error from fsck:

 fsck -fv -z ./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.undo.$(date
+"%Y-%m-%d.%H.%M.%S").und /dev/loop2
fsck from util-linux 2.29.2
e2fsck 1.43.4 (31-Jan-2017)
Overwriting existing filesystem; this can be undone using the command:
    e2undo ./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.undo.2019-01-17.13.31.41.und /dev/loop2

The filesystem size (according to the superblock) is 122063840 blocks
The physical size of the device is 121604515 blocks
Either the superblock or the partition table is likely to be corrupt!

Info from fdisk -l /dev/sda

Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000794ac

Device     Boot  Start       End   Sectors   Size Id Type
/dev/sda1  *        32      8191      8160     4M  4 FAT16 <32M
/dev/sda2       262144 976773119 976510976 465.7G 83 Linux

Info from fdisk -l ./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.img

Disk ./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.img: 464 GiB, 498226311168 bytes, 973098264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000794ac

Device                                              Boot  Start       End   Sectors   Size Id Type
./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.img1 *        32      8191      8160     4M  4 FAT16 <32M
./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.img2      262144 976773119 976510976 465.7G 83 Linux

I made a loop device for the Partition Using:

losetup --offset $((512*262144)) /dev/loop2 ./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.img

From blockdev --getbsz /dev/loop2

4096

From blockdev --getsz /dev/loop2

972836120

From dumpe2fs /dev/loop2:

Filesystem UUID:          f68ccb5a-bcfa-4e8a-8876-45adaa6e6b85
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              30523392
Block count:              122063840
Reserved block count:     6103192
Free blocks:              96939245
Free inodes:              30462657
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      994
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Sat Apr 26 21:28:22 2014
Last mount time:          Wed Jan 16 15:59:22 2019
Last write time:          Thu Jan 17 18:16:50 2019
Mount count:              17
Maximum mount count:      -1
Last checked:             Sat Apr 26 21:28:22 2014
Check interval:           0 (<none>)
Lifetime writes:          10 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      162d0daa-7968-48f9-8370-f095c9e19f58
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             128M
Journal length:           32768
Journal sequence:         0x000059bd
Journal start:            0

Followed by a lot of:

Group 0: (Blocks 0-32767)
  Primary superblock at 0, Group descriptors at 1-30
  Reserved GDT blocks at 31-1024
  Block bitmap at 1025 (+1025)
  Inode bitmap at 1026 (+1026)
  Inode table at 1027-1538 (+1027)
  4 free blocks, 8179 free inodes, 2 directories
...
(SKIPPING TO END)
...
Group 3725: (Blocks 122060800-122063839)
  Block bitmap at 122060800 (+0)
  Inode bitmap at 122060801 (+1)
  Inode table at 122060802-122061313 (+2)
  0 free blocks, 8192 free inodes, 0 directories

Finished with:

dumpe2fs: /dev/loop2: error reading bitmaps: Can't read a block bitmap

Now I can mount /dev/sda2 just fine and read the files I cannot however mount /dev/loop2

mount -t ext3 /dev/loop2 ./DriveImage/
mount: wrong fs type, bad option, bad superblock on /dev/loop2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

I get the same error trying to mount directly from the image using:

mount -o loop,offset=$((512*262144)) ./Seagate.ST3500320NS.SN-9QM5ZHHR.500GB.465GiB.img ./DriveImage

Now According to dumpe2fs the superblock is right!
And According to the math:

 Superblock says:
 122063840
 Filesystem says:
 121604515

 block size:
 4096

 Math: Sectors * Sector Size = Size / Block Size = Blocks
 Partition 1: 
 8160 * 512 = 4177920 / 4096 = 1020
 Partition 2:
 [From fdisk]
 976510976 * 512 = 499973619712 / 4096 = 122063872
 [From blockdev with /dev/loop2]
 972836120 * 512 = 498092093440 / 4096 = 121604515

fdisk is reporting pretty close to the right blocks...(Just 32 more blocks) But in my book, fsck is getting its blocksize info the same way blockdev does (or uses it), but according to dumpe2fs and checking the actual partition table, the superblock is actually right and so is the partition table.

In fear of losing the original data on the disk (10 years worth of family pictures/videos and important files) I am not willing to run this stuff on the original disk. So I made a copy of the disk to this image, then also, in case I screw up something, I also made a copy of the image. (Don't worry, I have disk space for this).

What am I doing wrong here? and how can I fix this?

NOTE: Due to the old drive starting to fail (assumed, I have had some issues), this is why I am doing this.
ALSO, for some reason the drive lost its partition table and I had to use testdisk to recover it. After it was recovered I was able to mount the large partition and read all my data.
SO I assumed that either testdisk got it right or pretty darn close, since it is all there.

(UPDATE #1) I should also note, that when I run fsck on the original drive I do not get this error...

fsck -nfv /dev/sda2
fsck from util-linux 2.29.2
e2fsck 1.43.4 (31-Jan-2017)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

       60735 inodes used (0.20%, out of 30523392)
        1510 non-contiguous files (2.5%)
          49 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 23779/2425/0
    25124595 blocks used (20.58%, out of 122063840)
           0 bad blocks
           1 large file

       56022 regular files
        4704 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
       60726 files

(UPDATE #2) I found out that the image file is not the same as the drive, the image file is smaller.
Drive Size: 500107862016 (Had a mistake here, I only got the size of the second partition, updated to correct info)
Image Size: 498226311168
The Image file is missing 1881550848 bytes, over 1.88GB of data. (This corrected also)
Seems like dd did not get everything, and that I might be right that the drive has problems, is there a way to make dd fill read-errors with blank space so I can have a matching size?

I am running through fsck on the loop device to see what it does, I guess if I mess it up, I will just restore the back image.

ANOTHER IMPORTANT NOTE: This is a headless server system, no GUI, just CLI.

Solution

Errors from a bad disk image

Sounds like you're seeing errors in the image cause by a failing drive. I'd use gddrescue instead, it attempts to handle read errors.

Gddrescue's manual is informative, it's section 10 A small tutorial with examples starts with

Example 1: Fully automatic rescue of a whole disc with two ext2 partitions in /dev/sda to /dev/sdb.
Note: you don't need to partition /dev/sdb beforehand, but if the partition table on /dev/sda is damaged, you'll need to recreate it somehow on /dev/sdb.

ddrescue -f -r3 /dev/sda /dev/sdb mapfile
fdisk /dev/sdb
e2fsck -v -f /dev/sdb1
e2fsck -v -f /dev/sdb2

Instead of rescuing directly to a device (/dev/sdb), using a file works. And instead of starting with -r 3 to retry bad sectors 3 times, maybe use the default (0) and -n / --no-scrape to "Skip the scraping phase" to get as much as you can quickly first.

There's also a ddrescueview package that gives a graphic view of a gddrescue map file, that might be interesting:

And monitoring syslog or dmesg should have shown read errors earlier, I'd monitor them while using that drive.

Are there that many important files to backup?

If the files are still readable, and especially if the important files you want to backup are much smaller than the entire drive, just copy those files only and forget about the entire drive image. OS's are easy to reinstall.

Mounting full disk images

Looks like losetup -P creates appropriate partition-loop devices itself, or maybe partprobe or gnome-disk-image-mounter, in addition to kpartx.

Comments

[AJ Smith 'Smugger']

I did find some info saying stuff on doing a filesystem resize might help [resize2fs] but due to it taking almost 3 hours to make a copy of the image file, screw ups would cost me tons of time.. Would this be the right route. Sources: access.redhat.com/solutions/55010 ; wiki.vpsget.com/index.php/…

[Xen2050]

I'm seeing /dev/sda2, /dev/loop2, an image file mounted with offsets... is the image of the entire sda drive? There are utilities to automatically create loop devices for each partition without needing to do the math, I think kpartx still works, and I'm sure there's newer ones. Using one of those might give different results. And is the image accurate, a diff or hash matches?

[AJ Smith 'Smugger']

Yes, the fdisk results show you the partition table of both sda and the image file, yes I copied the entire drive, not just the single partition, as I needed to keep the entire drive data, just in case testdisk got the partition table wrong. Which, I assume it did not since I am able to mount the original drive partition without a problem and fsck returns no errors.

[AJ Smith 'Smugger']

kpartx is not part of the base linux system tools

[davidgo]

@Xen2050 is correct, using an offset likely causes problems and does not help you. If you want to backup a disk use /dev/sdx, if you want to back up a partition use /dev/sdx?

[AJ Smith 'Smugger']

As you did say something about the diff/hash, I was already checking this out just before I had to step out of the house, I was going to post and update on this info, seems like (and I did not know this) when dd runs if it can not read a sector it skips it but does not leave any empty data, so you are right, the size of the image is smaller than the drive. I will post an update about this...

[AJ Smith 'Smugger']

@davidgo I need the entire drive

[AJ Smith 'Smugger']

But just for sake of argument, I will also do a partition backup, this will take a while.

[AJ Smith 'Smugger']

I had a mistake in the drive size update that I did, now fixed

[AJ Smith 'Smugger']

I should probably have mentioned that this is a headless server system I am working under. No GUI, only CLI. As for the gddrescue, I just came across a forum talking about using ddrescue (the non-gui) and not dd to fill in the gaps, since one of the things is dd is not doing that. I am going to try that and see if that works.

[AJ Smith 'Smugger']

And Yes, many important files. and no, it does not have an os, just files.

[AJ Smith 'Smugger']

If ddresue works for my case, I would be willing to accept this answer if you update it to fit my case scenario of only CLI. Could you also put in some example commands on how to use ddrescue along with the link to the man page, I personally know, but for future visitors.

[Xen2050]

There is an older different program also called ddrescue (it's binary is dd_rescue) that I wouldn't use. Confusingly the gddrescue package's binary is also called ddrescue. gddrescue is a terminal program, along with losetup, kpartx, gnome-disk-image-mounter (but that one's part of gnome-disk-utility that does have a GUI). (Also potentially confusing, when losetup -P creates partition loops (like loop0p1) they're not listed by losetup -a (at least in Debian stable they're not).)

[AJ Smith 'Smugger']

Oh, when I looked up gddrescue it showed it to be GUI based. My mistake, I will check it out.

[AJ Smith 'Smugger']

Awesome update, I am going to accept this, as I am almost fairly sure this is the problem that dd did not get all the data and missed some, thank you very much for your help!

[Xen2050]

Welcome :) It gets tricky comparing block sizes & sectors & image sizes in bytes, and I learned that even mounting a disk as read-only (ro) might still write to the device, changing it's hash & making an image no longer identical. (see man mount for -r and noload, also blockdev --setro). -- By far the best advice is: Have backups now, so you don't need to rescue data later!