Re: Raid6 array crashed-- 4-disk failure...(?)

[Maarten <maarten@ultratux.net>]

Peter Grandi wrote:
>> This weekend I promoted my new 6-disk raid6 array to
>> production use and was busy copying data to it overnight. The
>> next morning the machine had crashed, and the array is down
>> with an (apparent?) 4-disk failure, [ ... ]
> 
> Multiple drive failures are far more common than people expect,
> and the problem lies in people's expectations, because they don't
> do common mode analysis (what's what? many will think).

It IS more common indeed. I'm on my seventh or eight raid-5 array now, 
the first was a 4-disk raid5 40(120) GB array. I've had 4 or 5 two-disk 
failures happen to me over the years, invariably during rebuild, indeed.
This is why I'm switching over to raid-6, by the way.

I did not, at any point, lose the array with the two-disk failures 
though. I intelligently cloned bad drives with dd_rescue and reassembled 
those degraded arrays using the new disks and thus got my data back.
But still, such events tend to keep me busy for a whole weekend, which 
is not too pleasant.

> They typically happen all at once at power up, or in short
> succession (e.g. 2nd drive fails while syncing to recover from
> 1st failure).
> 
> The typical RAID has N drives from the same manufacturer, of the
> same model, with nearly contiguous serial numbers, from the same
> shipping carton, in an enclosure where they all are started and
> stopped at the same time, run on the same power circuit, at the
> same temperature, on much the same load, attached to the same
> host adapter or N of the same type. Expecting as many do to have
> uncorrelated failures is rather comical.

This is true. However, since I know this fact I tend to take care to not 
make it too vulnerable; the system is incredibly well cooled, it has 8 
80mm fans that cool the 16(!) disks, I buy disks in batches of 2, from 
different brands and vendors. It indeed has just one PSU, but I chose a 
good one, I think it's a Tagan 550 Watt unit.

In fact -this is my home system- since I cannot afford a DLT drive for 
this much data I practically have no backup, so I really spend a lot of 
effort making sure the array stays ok. Yes, I know, this not a good 
idea, but how do I economically backup 3 TB ?
In practice I have older disks and/or decommisioned arrays with 
"backups" but this is of course not up to date at all.

> 1) Is my analysis correct so far ?
> 
> Not so sure :-). Consider this interesting discrepancy:
> 
>   /dev/sda1:
>   [ ... ]
>       Raid Devices : 7
>      Total Devices : 6
>   [ ... ]
>     Active Devices : 5
>   Working Devices : 5
> 
>   /dev/sdb1:
>   [ ... ]
>       Raid Devices : 7
>      Total Devices : 6
>   [ ... ]
>     Active Devices : 6
>   Working Devices : 6
> 
> Also note that member 0, 'sdk1' is listed as "removed", but not
> faulty, in some member statuses. However you have been able to
> actually get the status out of all members, including 'sdk1',
> which reports itself as 'active', like all other drives as of
> 5:16. Then only 2 drives report themselves as 'active' as of
> 5:17, and those think that the array has 5 'active'/'working'
> devices at that time. What happened between 5:16 and 5:17?

Don't know, I was asleep ;-)
Seriously, the system experienced a hard crash. Not even the keyboard 
responded to the capslock key/led anymore. Logs are empty.

> You should look at your system log to figure out what really
> happened to your drives and then assess what the cause of the
> failure was and its impact.

Syslogs are empty. Not one line nor even a hint at that time.

> 3) Should I say farewell to my ~2400 GB of data ? :-(
> 
> Surely not -- you have a backup of those 2400GB, as obvious from
> "busy copying data to it". RAID is not backup anyhow :-).

Yes I have most of the data. What I'd lose is ~20 GB, which is less than 
one percent ;-).  But still, it's a lot of bytes...

> 4) If it was only a one-drive failure, why did it kill the array ?
> 
> The MD subsystem marked as bad more than one drive. Anyhow doing
> a 5+2 RAID6 and then loading it with data with a checksum drive
> missing and at the same time as it syncing seems a bit too clever
> to me. Right now the array is running in effect in RAID0 mode, so
> I would not trust it even if you are able to restart it.

Just bought a seventh/replacement disk...  But if the array is lost that 
is of little use.  I'll try to reassemble later tonight...

Thanks,
Maarten

> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html