Kernel panic on Linux Red Hat server

Question

Linux server has been running fine, no updates installed recently, but out of the blue this morning the server stopped responding. I went to check it and it was saying there is a problem with the filesystem, and stated 'you should run fsck manually'.

I ran fsck on the relevant partition and fix all the inode problems suggested. After restarting I am now shown the following messages:

JBD: no valid journal superblock found
EXT3-fs: error loading journal
mount: error 22 mounting ext3
Switching to new root
ERROR opening /dev/console!!!!:2
Kernel panic - no syncing: Attempting to kill init

Does anyone have any idea how I can resolve this? I am not particularly familiar with Linux so please be verbose when suggestions solutions.

I don't want to reinstall Linux if I can avoid it as the server has about 5 years worth of valuable data on there. Any help appreciated!

Solution 1

As @Blender mentioned in his comment this sounds like a hard drive failure. As you don't know Linux very well, I would suggest your first port of call would be to find someone who does as fixing this sort of issue (or even understanding fully) can be hard if you don't know some of the intricacies of Linux.

First use either a Live CD or Live USB stick (see Knoppix as an example or Ubuntu) to boot the machine and see if it can mount the disk(s). If it can, copy the important data off (you should be backing this stuff up anyway but that is a separate issue) and then try and track down what hardware failed. If it is an hard disk then replace it.

You can also look at the logs to try and track down what the issue was in the first place. For hardware I'd check /var/log/dmesg. Depending on the type of drive (and version of the kernel etc.) you'll see different messages. When the drive is connected to a SCSI/Raid controller you may see messages relating to SCSI commands being corrupt or not being responded to. You may also see messages saying writes have timed out (as another example). It is hard to say what you might see as it depends on hardware and kernel versions.

You can use smartctl --all /dev/<hard disk> to check on the SMART information. This is built into the hard drive and is the way that the disk can tell you how sound it is.

The JBD error you see is because the EXT3 journal is mangled. If the Live USB/CD distro can't mount the partition because of this you can mount ext3 partitions as ext2 (which ignores the journal) to get the data off. But again, this is something you may need to be a little more seasoned at using Unix to be able to pull off. However, you can do a Google for this procedure and see if you can perform it while booted off the Live USB/CD to allow you to mount the partition.

Solution 2

Boot with a rescue cd and check the disks.

Solution 3

As Jure1873 suggested, get a boot disc and fsck the drive.

But while you're at it, mount the disk and see if you can read the logs that are recorded. Look for information leading to disk failure. Heck, see if the boot disc also shows signs in the logs of errors as it reads from the drive.

I'd also try copying the data off the disk to another disk, using something like dd-rescue, if the data is valuable to you.

Comments

[Kyle Smith]

Do you have backups?

[ninjalj]

First, copy as much as you can to another disk. If the drive is failing, some data may be unrecoverable soon.