kernel: scsi 0:0:0:0: rejecting I/O to dead device

Question

Yesterday, OSSEC sent me an alerting email:

Jul 29 21:25:16 SVR4149 kernel: end_request: I/O error, dev sda, sector 334634969
Jul 29 21:25:16 SVR4149 kernel: sd 0:0:0:0: SCSI error: return code = 0x00040000
Jul 29 21:25:16 SVR4149 kernel: end_request: I/O error, dev sda, sector 334634977
Jul 29 21:28:28 SVR4149 kernel: sd 0:0:0:0: SCSI error: return code = 0x00040000

The surprising is at that time I have only /dev/sdb device.

# fdisk -l

Disk /dev/sdb: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   83  Linux
/dev/sdb2              14        7662    61440592+  83  Linux
/dev/sdb3            7663        8706     8385930   82  Linux swap / Solaris
/dev/sdb4            8707       38888   242436915    5  Extended
/dev/sdb5            8707       38888   242436883+  83  Linux

After Googling, I found this link. Do the suggested commands, it bring back my /dev/sdc:

Jul 29 22:55:45 SVR4149 kernel: ata1: hard resetting link
Jul 29 22:55:45 SVR4149 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jul 29 22:55:45 SVR4149 kernel: ata1.00: ATA-8: WDC WD3202ABYS-01B7A0, 02.03B02, max UDMA/133
Jul 29 22:55:45 SVR4149 kernel: ata1.00: 625142448 sectors, multi 0: LBA48 NCQ (depth 31/32)
Jul 29 22:55:45 SVR4149 kernel: ata1.00: configured for UDMA/133
Jul 29 22:55:45 SVR4149 kernel: ata1: EH complete
Jul 29 22:55:45 SVR4149 kernel: ata1.00: detaching (SCSI 0:0:0:0)
Jul 29 22:55:45 SVR4149 kernel:   Vendor: ATA       Model: WDC WD3202ABYS-0  Rev: 02.0
Jul 29 22:55:45 SVR4149 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jul 29 22:55:45 SVR4149 kernel: SCSI device sdc: 625142448 512-byte hdwr sectors (320073 MB)
Jul 29 22:55:45 SVR4149 kernel: sdc: Write Protect is off
Jul 29 22:55:45 SVR4149 kernel: sdc: Mode Sense: 00 3a 00 00
Jul 29 22:55:45 SVR4149 kernel: SCSI device sdc: drive cache: write back
Jul 29 22:55:53 SVR4149 kernel: SCSI device sdc: 625142448 512-byte hdwr sectors (320073 MB)
Jul 29 22:55:53 SVR4149 kernel:  sdc: sdc1 sdc2 sdc3 sdc4 < sdc5 >
Jul 29 22:55:53 SVR4149 kernel: sd 0:0:0:0: Attached scsi disk sdc
Jul 29 22:55:53 SVR4149 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0

Rechecking with fdisk:

# fdisk -l

Disk /dev/sdb: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   83  Linux
/dev/sdb2              14        7662    61440592+  83  Linux
/dev/sdb3            7663        8706     8385930   82  Linux swap / Solaris
/dev/sdb4            8707       38888   242436915    5  Extended
/dev/sdb5            8707       38888   242436883+  83  Linux

Disk /dev/sdc: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *           1          13      104391   83  Linux
/dev/sdc2              14        7662    61440592+  83  Linux
/dev/sdc3            7663        8706     8385930   82  Linux swap / Solaris
/dev/sdc4            8707       38888   242436915    5  Extended
/dev/sdc5            8707       38888   242436883+  83  Linux

But I got the another problem from kernel log:

Jul 30 01:03:41 SVR4149 kernel: scsi 0:0:0:0: rejecting I/O to dead device
Jul 30 01:14:40 SVR4149 kernel: scsi 0:0:0:0: rejecting I/O to dead device
Jul 30 01:16:41 SVR4149 kernel: scsi 0:0:0:0: rejecting I/O to dead device
Jul 30 01:53:18 SVR4149 last message repeated 7 times

and smartd keep opening the non exist device:

Jul 30 10:00:57 SVR4149 smartd[3749]: Device: /dev/sda, No such device, open() failed

No special in my smartd.conf file:

# grep -v "^#" /etc/smartd.conf | sed '/^$/d'
DEVICESCAN -H -m root

Does my scsi0 is going to 'die'?

cat /proc/scsi/scsi 
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: WDC WD3202ABYS-0 Rev: 02.0
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: WDC WD3202ABYS-0 Rev: 02.0
  Type:   Direct-Access                    ANSI SCSI revision: 05

Any help will be appreciated.

Solution

Looks like the drive is dropping and then reconnecting. This indicates one of three things:

1. Most likely a bad drive, I would start checking the SMART logs and see what you get there.
2. A bad cable/SCSI controller (normally RAID cards) ... if SMART checks out and this continues, swap the cable first then the card.
3. You are doing so much sustained disk I/O that you are overloading the disk controller ... you should be able to tell if you are overloading the I/O.

Hope that helps ... its a scary message to get.

Comments

[Coops]

I got this after yet another power failure overnight and we have no UPS at current. Think this will be a kick in the balls to get one!