mdadm --add Device or resource busy error and multiple drives "failing" on RAID 6 at once

mdadm --add Device or resource busy error and multiple drives "failing" on RAID 6 at once

[fibreraid]

Hi,

I am using Ubuntu Jaunty (9.04) 64-bit with mdadm 2.6.9 on a whitebox
server with 8GB RAM and dual Xeon quad-core CPUs. I have a RAID 6
consisting of 12 drives all located in the same enclosure, using the
1.2 superblock. For some reason, mdadm seems to "kick out" numerous
drives out of the array at once (sometimes 5 or more). This results in
a degraded state. I've tried replacing the cabling, SAS controller to
which the drives are connected, and even updating the drive firmware
but mdadm still seems to do this on occasion (like once a week). I am
highly doubting all these drives are failing at once. Any further
ideas on what might be causing this issue?

Also, when I attempt to add the kicked out drives (their partitions)
back in using mdadm --add -f (to force it to clear the failed flag), I
often get a "Device or resource busy error". Is there any way around
this error, without rebooting the machine? If I reboot the machine,
then I can easily using the -add -f command, but I would rather not
reboot.

I welcome your ideas. Thank you for your guidance!

-Steve S

Re: mdadm --add Device or resource busy error and multiple drives "failing" on RAID 6 at once

[John Robinson]

On 16/07/2010 12:54, fibreraid@gmail.com wrote:
> I am using Ubuntu Jaunty (9.04) 64-bit with mdadm 2.6.9 on a whitebox
> server with 8GB RAM and dual Xeon quad-core CPUs. I have a RAID 6
> consisting of 12 drives all located in the same enclosure, using the
> 1.2 superblock. For some reason, mdadm seems to "kick out" numerous
> drives out of the array at once (sometimes 5 or more). This results in
> a degraded state. I've tried replacing the cabling, SAS controller to
> which the drives are connected, and even updating the drive firmware
> but mdadm still seems to do this on occasion (like once a week). I am
> highly doubting all these drives are failing at once. Any further
> ideas on what might be causing this issue?

I recognise these symptoms from earlier postings, relating to problems 
with the driver for a controller card/chip (I forget which one) and 
S.M.A.R.T. - you might grep the list archives for references to your 
controller.

Cheers,

John.

Re: mdadm --add Device or resource busy error and multiple drives "failing" on RAID 6 at once

[Richard]

John Robinson wrote:

> I recognise these symptoms from earlier postings, relating to problems 
> with the driver for a controller card/chip (I forget which one) and 
> S.M.A.R.T. - you might grep the list archives for references to your 
> controller.

LSI 1068 based controllers.

Regards,

Richard