Re: raid5: Disk failure on sdm, disabling device

["David M. Strang" <dstrang@shellpower.net>]

Forrest Taylor wrote:
> > Michael Tokarev wrote:
> > > David M. Strang wrote:
> > > []
> > >
> > > > Is there something a little deeper to this error message?
> > > >
> > > > Aug 31 04:48:15 abyss kernel: scsi2 (12:0): rejecting I/O to offline
> > > > device
> > > > Aug 31 04:48:15 abyss kernel: raid5: Disk failure on sdm, disabling
> > > > device.
> > >
> > > If you reread my message, I hope you will find a bit of clue:
> > >
> > >   echo y > /sys/block/sdm/device/delete
> > > (or something like that anyway.)
> > >
> > > That's needed for exactly this purpose: to remove a device which is
> > > marked 'offline' in the kernel somewhere - to be able to re-add it
> > > later - either with this script or just doing that magic 
> > > add-single-device
> > > command manually.
> >
> > I should have listened sooner; I wasn't able to follow the paths in the
> > re-add command, so I didn't do the delete.
> >
> > I did the delete (the above syntax is correct) -- ran the rescan script, 
> > it
> > added it back - and the device is accessable again.
> >
> > Thank you Michael & Forrest.
>
> I just noticed that in /sys/block/sda/device/, along with the delete
> file, there is a rescan file.  I think that this is what I was thinking
> of with the 2.6 kernel.

Well, my worst fear was realized. My raid array barfed during rebuild. It 
errorenously kicked a few devices from the array. I/O was not rejected to 
any of the drives as was before; so I stopped the array. (mdadm --stop 
/dev/md0) -- and re-assembled it;

mdadm -A /dev/md0 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf 
/dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn 
/dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv 
/dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab -f

Just as before;

mdadm: forcing event count in /dev/sda(0) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdb(1) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdc(2) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdd(3) from 337384 upto 2645518
mdadm: forcing event count in /dev/sde(4) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdf(5) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdg(6) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdh(7) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdi(8) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdj(9) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdk(10) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdl(11) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdn(13) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdo(14) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdp(15) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdq(16) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdr(17) from 337384 upto 2645518
mdadm: forcing event count in /dev/sds(18) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdt(19) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdu(20) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdv(21) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdw(22) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdx(23) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdy(24) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdz(25) from 337384 upto 2645518
mdadm: forcing event count in /dev/sdab(27) from 337384 upto 2645518
mdadm: /dev/md0 assembled from 26 drives - not enough to start the array.

The 2 drives, one of which was sync'd -- and one that was rebuilding; won't 
join the array.