raid5: Disk failure on sdm, disabling device

raid5: Disk failure on sdm, disabling device

[David M. Strang]

Okay, my array is degraded -- and the device /dev/sdm is disabled.

Short of a reboot, is there a way to re-enable the device? It's already 
flagged as faulty in mdadm.

-- David M. Strang 

Re: raid5: Disk failure on sdm, disabling device

[Forrest Taylor]

On Wed, 2005-08-31 at 13:26, David M. Strang wrote:
> Okay, my array is degraded -- and the device /dev/sdm is disabled.
> 
> Short of a reboot, is there a way to re-enable the device? It's already 
> flagged as faulty in mdadm.

Have you tried to remove /dev/sdm from the RAID and add it back in?  I
guess that the larger question is what caused it to be flagged faulty?

Forrest

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

Aug 31 04:48:15 abyss kernel: md: excessive errors occurred during 
superblock update, exiting
Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline device
Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling device. 
Operation continuing on 27 devices

However, in the past -- I had to rebuild the array; and now it looks like 
this:

/dev/md0:
        Version : 01.00.01
  Creation Time : Wed Dec 31 19:00:00 1969
     Raid Level : raid5
     Array Size : 1935556992 (1845.89 GiB 1982.01 GB)
    Device Size : 71687296 (68.37 GiB 73.41 GB)
   Raid Devices : 28
  Total Devices : 28
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Aug 31 11:04:22 2005
          State : clean, degraded
 Active Devices : 27
Working Devices : 27
 Failed Devices : 1
  Spare Devices : 0

         Layout : left-asymmetric
     Chunk Size : 128K

           UUID : 4e2b6b0a8e:92e91c0c:018a4bf0:9bb74d
         Events : 333051

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync 
/dev/scsi/host2/bus0/target0/lun0/disc
       1       8       16        1      active sync 
/dev/scsi/host2/bus0/target1/lun0/disc
       2       8       32        2      active sync 
/dev/scsi/host2/bus0/target2/lun0/disc
       3       8       48        3      active sync 
/dev/scsi/host2/bus0/target3/lun0/disc
       4       8       64        4      active sync 
/dev/scsi/host2/bus0/target4/lun0/disc
       5       8       80        5      active sync 
/dev/scsi/host2/bus0/target5/lun0/disc
       6       8       96        6      active sync 
/dev/scsi/host2/bus0/target6/lun0/disc
       7       8      112        7      active sync 
/dev/scsi/host2/bus0/target7/lun0/disc
       8       8      128        8      active sync 
/dev/scsi/host2/bus0/target8/lun0/disc
       9       8      144        9      active sync 
/dev/scsi/host2/bus0/target9/lun0/disc
      10       8      160       10      active sync 
/dev/scsi/host2/bus0/target10/lun0/disc
      11       8      176       11      active sync 
/dev/scsi/host2/bus0/target11/lun0/disc
      12       8      192        -      faulty 
/dev/scsi/host2/bus0/target12/lun0/disc
      13       8      208       13      active sync 
/dev/scsi/host2/bus0/target13/lun0/disc
      14       8      224       14      active sync 
/dev/scsi/host2/bus0/target14/lun0/disc
      15       8      240       15      active sync 
/dev/scsi/host2/bus0/target15/lun0/disc
      16      65        0       16      active sync 
/dev/scsi/host2/bus0/target16/lun0/disc
      17      65       16       17      active sync 
/dev/scsi/host2/bus0/target17/lun0/disc
      18      65       32       18      active sync 
/dev/scsi/host2/bus0/target18/lun0/disc
      19      65       48       19      active sync 
/dev/scsi/host2/bus0/target19/lun0/disc
      20      65       64       20      active sync 
/dev/scsi/host2/bus0/target20/lun0/disc
      21      65       80       21      active sync 
/dev/scsi/host2/bus0/target21/lun0/disc
      22      65       96       22      active sync 
/dev/scsi/host2/bus0/target22/lun0/disc
      23      65      112       23      active sync 
/dev/scsi/host2/bus0/target23/lun0/disc
      24      65      128       24      active sync 
/dev/scsi/host2/bus0/target24/lun0/disc
      25      65      144       25      active sync 
/dev/scsi/host2/bus0/target25/lun0/disc
      26       0        0        -      removed
      27      65      176       27      active sync 
/dev/scsi/host2/bus0/target27/lun0/disc

      28      65      160       26      active sync 
/dev/scsi/host2/bus0/target26/lun0/disc


My last reboot -- which was due in part to a power failure; this device --  
/dev/scsi/host2/bus0/target26/lun0/disc -- wouldn't join back into the raid 
w/o being re-hotadded. With another device failed, I'm fearful of a reboot. 
I want to attempt to rebuild the failed device; and see what happens.

-- David M. Strang



----- Original Message ----- 
From: Forrest Taylor
To: David M. Strang
Cc: Linux RAID
Sent: Wednesday, August 31, 2005 3:28 PM
Subject: Re: raid5: Disk failure on sdm, disabling device


On Wed, 2005-08-31 at 13:26, David M. Strang wrote:
> Okay, my array is degraded -- and the device /dev/sdm is disabled.
>
> Short of a reboot, is there a way to re-enable the device? It's already
> flagged as faulty in mdadm.

Have you tried to remove /dev/sdm from the RAID and add it back in?  I
guess that the larger question is what caused it to be flagged faulty?

Forrest

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html 

Re: raid5: Disk failure on sdm, disabling device

[Michael Tokarev]

David M. Strang wrote:
> Okay, my array is degraded -- and the device /dev/sdm is disabled.
> 
> Short of a reboot, is there a way to re-enable the device? It's already 
> flagged as faulty in mdadm.

man mdadm
mdadm --remove /dev/mdN /dev/sdm
mdadm --add    /dev/mdN /dev/sdm

/mjt

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

mdadm --remove /dev/md0 /dev/sdm
mdadm: hot removed /dev/sdm
mdadm --add /dev/md0 /dev/sdm
mdadm: Cannot open /dev/sdm: No such device or address

The device is disabled at a 'hardware?' level -- I can't even cfdisk 
/dev/sdm

----- Original Message ----- 
From: Michael Tokarev
To: David M. Strang
Cc: linux-raid@vger.kernel.org
Sent: Wednesday, August 31, 2005 3:29 PM
Subject: Re: raid5: Disk failure on sdm, disabling device


David M. Strang wrote:
> Okay, my array is degraded -- and the device /dev/sdm is disabled.
>
> Short of a reboot, is there a way to re-enable the device? It's already 
> flagged as faulty in mdadm.

man mdadm
mdadm --remove /dev/mdN /dev/sdm
mdadm --add    /dev/mdN /dev/sdm

/mjt 

Re: raid5: Disk failure on sdm, disabling device

[Michael Tokarev]

David M. Strang wrote:
> mdadm --remove /dev/md0 /dev/sdm
> mdadm: hot removed /dev/sdm
> mdadm --add /dev/md0 /dev/sdm
> mdadm: Cannot open /dev/sdm: No such device or address
> 
> The device is disabled at a 'hardware?' level -- I can't even cfdisk 
> /dev/sdm

Please don't top-post.

Well, you didn't mention that it's disappeared from the system
(as opposed to the raid array only).  And no, I for one don't
know how to re-add it - there are numerous reasons why it may
had disappeared, and it may be of different kinds of drive too
(like, SCSI or SATA).  Some cases can be handled (like, sending
"bus reset" and "start unit" commands followed by bus rescan
sometimes helps).  But it's safer to just reboot anyway (cold
reboot, with power off).

/mjt

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

Michael Tokarev  wrote:
> Please don't top-post.

My apologies; Outlook Express (/gasp) isn't the most condusive to mailing 
list replies.

> Well, you didn't mention that it's disappeared from the system
> (as opposed to the raid array only).  And no, I for one don't
> know how to re-add it - there are numerous reasons why it may
> had disappeared, and it may be of different kinds of drive too
> (like, SCSI or SATA).  Some cases can be handled (like, sending
> "bus reset" and "start unit" commands followed by bus rescan
> sometimes helps).  But it's safer to just reboot anyway (cold
> reboot, with power off).

It's a SCSI drive; in's a Dell 220F enclosure connected via a QLA2200 
adapter. I've pulled the bad disk (tho not 'yellow' at the hardware level; 
and re-inserted it -- but to no avail. It did cause a reset; but the device 
remains 'disabled'.

Aug 31 11:48:05 abyss kernel: qla2200 0000:00:09.0: LIP reset occured 
(f701).
Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LOOP DOWN detected.
Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LIP occured (f7f7).
Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LOOP UP detected (1 
Gbps).
Aug 31 11:48:11 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_abort 
scsi(2:0:6:0): cmd_timeout_in_sec=0x1e.
Aug 31 11:48:22 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_abort 
Exiting: status=Failed
Aug 31 11:48:22 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): DEVICE 
RESET ISSUED.
Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_device_reset: 
failed while waiting for commands
Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): LOOP 
RESET ISSUED.
Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: LIP reset occured 
(f701).
Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: LIP occured (f7f7).
Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_bus_reset: 
reset failed
Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): ADAPTER 
RESET issued.
Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: Performing ISP error 
recovery - ha= c21881cc.
Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: LIP reset occured 
(f8f7).
Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: LIP occured (f8f7).
Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: LOOP UP detected (1 
Gbps).
Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_host_reset: 
reset succeded

-- David M. Strang 

Re: raid5: Disk failure on sdm, disabling device

[Michael Tokarev]

David M. Strang wrote:
[]
> It's a SCSI drive; in's a Dell 220F enclosure connected via a QLA2200 
> adapter. I've pulled the bad disk (tho not 'yellow' at the hardware 
> level; and re-inserted it -- but to no avail. It did cause a reset; but 
> the device remains 'disabled'.
> 
> Aug 31 11:48:05 abyss kernel: qla2200 0000:00:09.0: LIP reset occured (f701).
> Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LOOP DOWN detected.
> Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LIP occured (f7f7).
> Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LOOP UP detected (1 Gbps).
> Aug 31 11:48:11 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_abort scsi(2:0:6:0): cmd_timeout_in_sec=0x1e.
> Aug 31 11:48:22 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_abort Exiting: status=Failed
> Aug 31 11:48:22 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): DEVICE RESET ISSUED.
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_device_reset: failed while waiting for commands
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): LOOP RESET ISSUED.
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: LIP reset occured (f701).
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: LIP occured (f7f7).
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_bus_reset: reset failed
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): ADAPTER RESET issued.
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: Performing ISP error recovery - ha= c21881cc.
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: LIP reset occured (f8f7).
> Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: LIP occured (f8f7).
> Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: LOOP UP detected (1 Gbps).
> Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_host_reset: reset succeded

Ugh.
You'd better ask linux-scsi list about what does it mean.
Note linux does not really support scsi hotplug, at least not automatically.

Well, you can try to do
  echo -n 1 > /sys/block/sdm/device/delete
(or was it 'echo -n y' ?)
and either
   echo -n scsi add-single-device c h t l > /proc/scsi/scsi
(replacing c h t l with proper numbers), or
   echo y > /sys/bus/scsi_host/hostN/rescan

But I'm really not sure if it'll help - sometimes (with some driver combinations)
it helps, and sometimes does not.  I haven't used qla* at all.

/mjt

Re: raid5: Disk failure on sdm, disabling device

[Forrest Taylor]

On Wed, 2005-08-31 at 14:28, David M. Strang wrote:
> It's a SCSI drive; in's a Dell 220F enclosure connected via a QLA2200 
> adapter. I've pulled the bad disk (tho not 'yellow' at the hardware level; 
> and re-inserted it -- but to no avail. It did cause a reset; but the device 
> remains 'disabled'.
> 
> Aug 31 11:48:05 abyss kernel: qla2200 0000:00:09.0: LIP reset occured 
> (f701).
> Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LOOP DOWN detected.
> Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LIP occured (f7f7).
> Aug 31 11:48:07 abyss kernel: qla2200 0000:00:09.0: LOOP UP detected (1 
> Gbps).
> Aug 31 11:48:11 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_abort 
> scsi(2:0:6:0): cmd_timeout_in_sec=0x1e.
> Aug 31 11:48:22 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_abort 
> Exiting: status=Failed
> Aug 31 11:48:22 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): DEVICE 
> RESET ISSUED.
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_device_reset: 
> failed while waiting for commands
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): LOOP 
> RESET ISSUED.
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: LIP reset occured 
> (f701).
> Aug 31 11:48:33 abyss kernel: qla2200 0000:00:09.0: LIP occured (f7f7).
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_bus_reset: 
> reset failed
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: scsi(2:0:6:0): ADAPTER 
> RESET issued.
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: Performing ISP error 
> recovery - ha= c21881cc.
> Aug 31 11:48:44 abyss kernel: qla2200 0000:00:09.0: LIP reset occured 
> (f8f7).
> Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: LIP occured (f8f7).
> Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: LOOP UP detected (1 
> Gbps).
> Aug 31 11:48:45 abyss kernel: qla2200 0000:00:09.0: qla2xxx_eh_host_reset: 
> reset succeded

If you are running 2.4 kernel, did you try running rescan-scsi-bus.sh
(http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh)?

Forrest

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

On Wed, 2005-08-31 at 14:39, Forrest Taylor wrote:
> If you are running 2.4 kernel, did you try running rescan-scsi-bus.sh
> (http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh)?

uname -ar
Linux abyss 2.6.11.12 #2 SMP Thu Jul 21 07:49:40 UTC 2005 i686 unknown 
unknown GNU/Linux

Will it work on 2.6?

-- David 

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

On Wed, 2005-08-31 at 14:45, David M. Strang wrote:
>On Wed, 2005-08-31 at 14:39, Forrest Taylor wrote:
>> If you are running 2.4 kernel, did you try running rescan-scsi-bus.sh
>> (http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh)?

> uname -ar
> Linux abyss 2.6.11.12 #2 SMP Thu Jul 21 07:49:40 UTC 2005 i686 unknown 
> unknown GNU/Linux

> Will it work on 2.6?

http://bash.cyberciti.biz/diskadmin/rescan-scsi-bus.sh.html

Tried the above script...

Host adapter 0 (aic7xxx) found.
Host adapter 1 (aic7xxx) found.
Host adapter 2 (qla2xxx) found.
Scanning hosts  0 1 2 channels 0 for
 SCSI target IDs  0 1 2 3 4 5 6 7 , LUNs  0
Scanning for device 1 0 6 0 ...
OLD: Host: scsi1 Channel: 00 Id: 06 Lun: 00
      Vendor: HP       Model: Ultrium 1-SCSI   Rev: E34A
      Type:   Sequential-Access                ANSI SCSI revision: 03
Scanning for device 2 0 0 0 ...
OLD: Host: scsi2 Channel: 00 Id: 00 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 1 0 ...
OLD: Host: scsi2 Channel: 00 Id: 01 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 2 0 ...
OLD: Host: scsi2 Channel: 00 Id: 02 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 3 0 ...
OLD: Host: scsi2 Channel: 00 Id: 03 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 4 0 ...
OLD: Host: scsi2 Channel: 00 Id: 04 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 5 0 ...
OLD: Host: scsi2 Channel: 00 Id: 05 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 6 0 ...
OLD: Host: scsi2 Channel: 00 Id: 06 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 7 0 ...
OLD: Host: scsi2 Channel: 00 Id: 07 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
0 new device(s) found.
0 device(s) removed.


The output is odd... It only sees 7 of the 28 devices.

-- David

Re: raid5: Disk failure on sdm, disabling device

[Forrest Taylor]

On Wed, 2005-08-31 at 14:53, David M. Strang wrote:
> 
> The output is odd... It only sees 7 of the 28 devices.

It looks like the script only checks for devices 0-7
(idsearch=`seq 0 7`)

It looks like you can pass it a values for the number of devices:

./rescan-scsi-bus.sh -ids=27

See if that finds all devices.

Forrest

Re: raid5: Disk failure on sdm, disabling device

[Forrest Taylor]

On Wed, 2005-08-31 at 14:45, David M. Strang wrote:
> On Wed, 2005-08-31 at 14:39, Forrest Taylor wrote:
> > If you are running 2.4 kernel, did you try running rescan-scsi-bus.sh
> > (http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh)?
> 
> uname -ar
> Linux abyss 2.6.11.12 #2 SMP Thu Jul 21 07:49:40 UTC 2005 i686 unknown 
> unknown GNU/Linux
> 
> Will it work on 2.6?

I thought that it was integrated into 2.6 kernels (I guess that I was
wrong).  In the header of the file, Kurt says that he tested in on a
2.6.11 kernel, so I assume that it would work on a 2.6 kernel ;o)

Forrest

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

On Wed, 2005-08-31 at 14:55, Forrest Taylor wrote:
>On Wed, 2005-08-31 at 14:45, David M. Strang wrote:
>> > On Wed, 2005-08-31 at 14:39, Forrest Taylor wrote:
>> > If you are running 2.4 kernel, did you try running rescan-scsi-bus.sh
>> > (http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh)?
>>
>> uname -ar
>> Linux abyss 2.6.11.12 #2 SMP Thu Jul 21 07:49:40 UTC 2005 i686 unknown
>> unknown GNU/Linux
>>
>> Will it work on 2.6?
>
> I thought that it was integrated into 2.6 kernels (I guess that I was
> wrong).  In the header of the file, Kurt says that he tested in on a
> 2.6.11 kernel, so I assume that it would work on a 2.6 kernel ;o)

OK, well - ignore my moron attack on the previous reply; I adjusted it to 
scan for 30 ids, instead of just 7.

Host adapter 0 (aic7xxx) found.
Host adapter 1 (aic7xxx) found.
Host adapter 2 (qla2xxx) found.
Scanning hosts  0 1 2 channels 0 for
 SCSI target IDs  0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 
23 24 25 26 27 28 29 30 , LUNs  0
Scanning for device 1 0 6 0 ....
OLD: Host: scsi1 Channel: 00 Id: 06 Lun: 00
      Vendor: HP       Model: Ultrium 1-SCSI   Rev: E34A
      Type:   Sequential-Access                ANSI SCSI revision: 03
Scanning for device 2 0 0 0 ....
OLD: Host: scsi2 Channel: 00 Id: 00 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 1 0 ...
OLD: Host: scsi2 Channel: 00 Id: 01 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 2 0 ...
OLD: Host: scsi2 Channel: 00 Id: 02 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 3 0 ...
OLD: Host: scsi2 Channel: 00 Id: 03 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 4 0 ...
OLD: Host: scsi2 Channel: 00 Id: 04 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 5 0 ...
OLD: Host: scsi2 Channel: 00 Id: 05 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 6 0 ...
OLD: Host: scsi2 Channel: 00 Id: 06 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 7 0 ...
OLD: Host: scsi2 Channel: 00 Id: 07 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 8 0 ...
OLD: Host: scsi2 Channel: 00 Id: 08 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 9 0 ...
OLD: Host: scsi2 Channel: 00 Id: 09 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 10 0 ...
OLD: Host: scsi2 Channel: 00 Id: 10 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 11 0 ...
OLD: Host: scsi2 Channel: 00 Id: 11 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 12 0 ...
OLD: Host: scsi2 Channel: 00 Id: 12 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 13 0 ...
OLD: Host: scsi2 Channel: 00 Id: 13 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 14 0 ...
OLD: Host: scsi2 Channel: 00 Id: 14 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 15 0 ...
OLD: Host: scsi2 Channel: 00 Id: 15 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 16 0 ...
OLD: Host: scsi2 Channel: 00 Id: 16 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 17 0 ...
OLD: Host: scsi2 Channel: 00 Id: 17 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 18 0 ...
OLD: Host: scsi2 Channel: 00 Id: 18 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 19 0 ...
OLD: Host: scsi2 Channel: 00 Id: 19 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 20 0 ...
OLD: Host: scsi2 Channel: 00 Id: 20 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 21 0 ...
OLD: Host: scsi2 Channel: 00 Id: 21 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 22 0 ...
OLD: Host: scsi2 Channel: 00 Id: 22 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 23 0 ...
OLD: Host: scsi2 Channel: 00 Id: 23 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 24 0 ...
OLD: Host: scsi2 Channel: 00 Id: 24 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 25 0 ...
OLD: Host: scsi2 Channel: 00 Id: 25 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 26 0 ...
OLD: Host: scsi2 Channel: 00 Id: 26 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
Scanning for device 2 0 27 0 ...
OLD: Host: scsi2 Channel: 00 Id: 27 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03
0 new device(s) found.
0 device(s) removed.

It sees the device in question:

Scanning for device 2 0 12 0 ...
OLD: Host: scsi2 Channel: 00 Id: 12 Lun: 00
      Vendor: SEAGATE  Model: ST373405FC       Rev: DF00
      Type:   Direct-Access                    ANSI SCSI revision: 03


Is there something a little deeper to this error message?

Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline device
Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling device.

-- David

Re: raid5: Disk failure on sdm, disabling device

[Forrest Taylor]

On Wed, 2005-08-31 at 15:01, David M. Strang wrote:
> Is there something a little deeper to this error message?
> 
> Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline device
> Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling device.

This error would make me think that the disk is bad.  Have you tested
the disk, or replaced it with a known-good disk?

Forrest

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

>> Is there something a little deeper to this error message?
>>
>> Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline 
>> device
>> Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling 
>> device.
>
>This error would make me think that the disk is bad.  Have you tested
>the disk, or replaced it with a known-good disk?

I have removed the disk from the enclosure, and replaced it with a spare --  
however; the disk remains disabled within the OS.

-- David 

Re: raid5: Disk failure on sdm, disabling device

[Michael Tokarev]

David M. Strang wrote:
[]

> Is there something a little deeper to this error message?
> 
> Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline device
> Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling device.

If you reread my message, I hope you will find a bit of clue:

  echo y > /sys/block/sdm/device/delete
(or something like that anyway.)

That's needed for exactly this purpose: to remove a device which is
marked 'offline' in the kernel somewhere - to be able to re-add it
later - either with this script or just doing that magic add-single-device
command manually.

/mjt

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

----- Original Message ----- 
From: Michael Tokarev
To: David M. Strang
Cc: Linux RAID
Sent: Wednesday, August 31, 2005 5:29 PM
Subject: Re: raid5: Disk failure on sdm, disabling device

Michael Tokarev wrote:
> David M. Strang wrote:
> []
>
> > Is there something a little deeper to this error message?
> >
> > Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline 
> > device
> > Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling 
> > device.
>
> If you reread my message, I hope you will find a bit of clue:
>
>   echo y > /sys/block/sdm/device/delete
> (or something like that anyway.)
>
> That's needed for exactly this purpose: to remove a device which is
> marked 'offline' in the kernel somewhere - to be able to re-add it
> later - either with this script or just doing that magic add-single-device
> command manually.

I should have listened sooner; I wasn't able to follow the paths in the 
re-add command, so I didn't do the delete.

I did the delete (the above syntax is correct) -- ran the rescan script, it 
added it back - and the device is accessable again.

Thank you Michael & Forrest.

-- David 

Re: raid5: Disk failure on sdm, disabling device

[Forrest Taylor]

On Wed, 2005-08-31 at 15:34, David M. Strang wrote:
> ----- Original Message ----- 
> From: Michael Tokarev
> To: David M. Strang
> Cc: Linux RAID
> Sent: Wednesday, August 31, 2005 5:29 PM
> Subject: Re: raid5: Disk failure on sdm, disabling device
> 
> Michael Tokarev wrote:
> > David M. Strang wrote:
> > []
> >
> > > Is there something a little deeper to this error message?
> > >
> > > Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline 
> > > device
> > > Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling 
> > > device.
> >
> > If you reread my message, I hope you will find a bit of clue:
> >
> >   echo y > /sys/block/sdm/device/delete
> > (or something like that anyway.)
> >
> > That's needed for exactly this purpose: to remove a device which is
> > marked 'offline' in the kernel somewhere - to be able to re-add it
> > later - either with this script or just doing that magic add-single-device
> > command manually.
> 
> I should have listened sooner; I wasn't able to follow the paths in the 
> re-add command, so I didn't do the delete.
> 
> I did the delete (the above syntax is correct) -- ran the rescan script, it 
> added it back - and the device is accessable again.
> 
> Thank you Michael & Forrest.

I just noticed that in /sys/block/sda/device/, along with the delete
file, there is a rescan file.  I think that this is what I was thinking
of with the 2.6 kernel.

Forrest

Re: raid5: Disk failure on sdm, disabling device

[David M. Strang]

Forrest Taylor wrote:
> > Michael Tokarev wrote:
> > > David M. Strang wrote:
> > > []
> > >
> > > > Is there something a little deeper to this error message?
> > > >
> > > > Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline
> > > > device
> > > > Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling
> > > > device.
> > >
> > > If you reread my message, I hope you will find a bit of clue:
> > >
> > >   echo y > /sys/block/sdm/device/delete
> > > (or something like that anyway.)
> > >
> > > That's needed for exactly this purpose: to remove a device which is
> > > marked 'offline' in the kernel somewhere - to be able to re-add it
> > > later - either with this script or just doing that magic 
> > > add-single-device
> > > command manually.
> >
> > I should have listened sooner; I wasn't able to follow the paths in the
> > re-add command, so I didn't do the delete.
> >
> > I did the delete (the above syntax is correct) -- ran the rescan script, 
> > it
> > added it back - and the device is accessable again.
> >
> > Thank you Michael & Forrest.
>
> I just noticed that in /sys/block/sda/device/, along with the delete
> file, there is a rescan file.  I think that this is what I was thinking
> of with the 2.6 kernel.

Well, my worst fear was realized. My raid array barfed during rebuild. It 
errorenously kicked a few devices from the array. I/O was not rejected to 
any of the drives as was before; so I stopped the array. (mdadm --stop 
/dev/md0) -- and re-assembled it;

mdadm -A /dev/md0 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf 
/dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn 
/dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv 
/dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab -f

Just as before;

mdadm: forcing event count in /dev/sda(0) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdb(1) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdc(2) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdd(3) from 337384 upto 2645518
mdadm: forcing event count in /dev/sde(4) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdf(5) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdg(6) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdh(7) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdi(8) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdj(9) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdk(10) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdl(11) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdn(13) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdo(14) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdp(15) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdq(16) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdr(17) from 337384 upto 2645518
mdadm: forcing event count in /dev/sds(18) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdt(19) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdu(20) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdv(21) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdw(22) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdx(23) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdy(24) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdz(25) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdab(27) from 337384 upto 2645518
mdadm: /dev/md0 assembled from 26 drives - not enough to start the array.

The 2 drives, one of which was sync'd -- and one that was rebuilding; won't 
join the array.