How to recover data from a corrupted ext3 partition?

Question

A server of mine had a drive failure of some sort which caused the OS (CentOS 5) to crash and stop working (it refuses to boot).

So we put another drive with a working OS and from there we try to mount the partitions in the old drive.

Most partitions mount fine except for one: the /var partition, where my MySQL tables reside.
When I try to mount that one, I see these errors with dmesg:

sd 0:0:1:0: Unhandled sense code
sd 0:0:1:0: SCSI error: return code = 0x08100002
Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
sdb: Current: sense key: Medium Error
Add. Sense: Unrecovered read error

Info fld=0x4a47e
JBD: Failed to read block at offset 9863
JBD: recovery failed
EXT3-fs: error loading journal.

Is there a way I can recover the data in that partition?

---

EDIT:
As requested, the output of tune2fs -l /dev/sdb2 is:

tune2fs 1.39 (29-May-2006)
Filesystem volume name:   /var1
Last mounted on:          <not available>
Filesystem UUID:          d84f5181-24f3-40ce-9eaa-601ae5ae33bd
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              26214400
Block count:              26214063
Reserved block count:     1310703
Free blocks:              25127226
Free inodes:              26213665
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1017
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   1024
Filesystem created:       Thu May 13 18:14:28 2010
Last mount time:          Thu Nov 29 12:52:00 2012
Last write time:          Wed Mar 27 20:29:28 2013
Mount count:              15
Maximum mount count:      -1
Last checked:             Thu May 13 18:14:28 2010
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      35f38c48-3933-4c99-bde2-63b0eccf200d
Journal backup:           inode blocks

---

EDIT 2:
As suggested by @Hartmut, I run fsck.ext3 /dev/sdb2 with the following result:

e2fsck 1.39 (29-May-2006)
/var1: recovering journal
/var1: Attempt to read block from filesystem resulted in short read while reading block 11931

JBD: Failed to read block at offset 9863
fsck.ext3: No such device or address while trying to re-open /var1
e2fsck: io manager magic bad!

Solution 1

It appears that your hard drive has had a physical failure, and that it has affected a block containing the ext3 journal.

You will need a second blank hard drive, at least as large as the failed drive partition, to perform any sort of recovery of this disk. You will also need a destination to copy recovered files to, so let's call it a third blank hard drive, network file share, etc.

The general recovery process is going to be:

1. Copy the failed partition to the new drive using dd conv=noerror or better dd_rescue. This may take some time.

2. Perform all further operations on the copy Here I assume that you have copied /dev/sdb2 to /dev/sdc2 and that you will recover files to /dev/sdd2.

3. Since the journal is corrupt, we will remove it:

   tune2fs -O ^has_journal /dev/sdc2
   

4. Now complete an fsck of the device. This may take some time.

   e2fsck /dev/sdc2
   

5. Mount the filesystem read-only and attempt to recover files.

   mount -o ro /dev/sdc2 /mnt/baddrive
   mount /dev/sdd2 /mnt/recoveredfiles
   cp -av /mnt/baddrive/* /mnt/recoveredfiles
   

6. In no case should you ever use the original disk again. Replace it (under warranty, if it is still under warranty).

Solution 2

Did you try mounting it as ext2 filesystem with mount -t ext2 ... ? ext3 is backward compatible with ext2, so it should just ignore the journal that seems to be broken. It's not an ideal solution, but it may let you access some data if you're lucky!

Solution 3

It could be possible that superblocks of filesystems has been corrupted. you can follow below steps to recover superblocks.

# dumpe2fs /dev/sdb2 | grep -i superblock

Sample output :

Primary superblock at 0, Group descriptors at 1-6
Backup superblock at 32768, Group descriptors at 32769-32774
Backup superblock at 98304, Group descriptors at 98305-98310
Backup superblock at 163840, Group descriptors at 163841-163846
Backup superblock at 229376, Group descriptors at 229377-229382

Either you can fsck the partition with alternative superblock or you can mount the partition with alternative superblock without fsck on filesystem .

To check filesystem

# fsck.ext3 -b 32768 /dev/sda2

To mount filesystem with alternative superblock:

# mount sb={alternative-superblock} /dev/device /mnt
# mount sb=32768 /dev/sdb2 /mnt

and Try to browse files.

Comments

[Chinmay Moharana]

maybe this will help.

[GetFree]

I've just tried that but I get the error EXT2-fs: sdb2: couldn't mount because of unsupported optional features (4).

[etagenklo]

Please post the output of tune2fs -l /dev/sdb2

[GetFree]

@etagenklo, there, I updated the question.

[GetFree]

@Hartmut, I used the command mount -t ext2 /dev/sdb2 /mnt/sdb2. Also, I posted the results of fsck.ext3 in the question.

[Dennis Kaarsemaker]

I would make this: you need a second and third drive. And add step 1.5: copy this to the third drive so you can mess up the first copy and restore it from the second copy. You do not want to rely on being able to copy it off the bad disk again.

[Michael Hampton]

Good advice, but I also have the feeling that the OP doesn't have that many disks laying around. As always, caution is advised.

[etagenklo]

You can disable the filesystem journal using tune2fs -O ^has_journal /dev/sdb2. Afterwards, you can try to mount it again using mount -t ext2 /dev/sdb2 /mnt/sdb2 or alternatively try mounting it as ext3 again: mount /dev/sdb2 /mnt/sdb2.

[GetFree]

The drive disappeared from the list fdisk -l. I believe that happend after running fsck.ext3. I rebooted the machine but still the drive does not appear and I dont know how to make it appear again.

[GetFree]

What would be the most recent superblock backup? the first one in the list or the last one?

[Amit Biyani]

AFAIK, these all are same until whole HDD corrupt. Plus, recovery sometimes matters which filesystem are you using ext2, ext3 or ext4. SEE HERE

[Amit Biyani]

also you should try to run testdisk on particular filesystem to see further details

[GetFree]

The disk had ext3 partitions, but it's gone now, it suffered some kind of physical failure that damaged several sectors.

[kostix]

@GetFree, another way to mount a filesystem without touching its journal is to pass noload to mount, like this: mount -o ro,noload <dev> <mpoint>

[Nick Chapman]

It's important to note that before modifying the has_journal flag you might need to clear the needs_recovery flag.