Re: Failure propagation of concatenated raids ?

[Benjamin ESTRABAUD <ben.estrabaud@mpstor.com>]

On 15/06/16 10:49, Nicolas Noble wrote:
>> I
>> think in your case you're better off stopping an array that has less than
>> parity drives than it should, either using a udev rule or using mdadm
>> --monitor.
>
> I actually have been unsuccessful in these attempts so far. What
> happens is that you very quickly get processes that get indefinitely
> stuck (indefinitely as in 'waiting on a very very long kernel
> timeout') trying to write something, so that the ext4fs layer becomes
> unresponsive on these threads, or take a very long time. Killing the
> processes takes a very long time because they are stuck in a kernel
> operation. And if potentially more processes can spawn back up, the
> automated script starts an interesting game of whack-a-mole in order
> to unmount the filesystem.
>
> And you can't stop the underlying arrays without first stopping the
> whole chain (umount, stop the lvm volume, etc...), otherwise you
> simply get "device is busy" errors, hence the whack-a-mole process
> killing. The only working method I've managed to successfully
> implement is to programatically loop over the list of all the drives
> involved in the filesystem, on all the raids involved, and flag all of
> them as failed drives. This way, you get to really put "emergency
> brakes" on. I find that to be a very, very scary method however.
>
I understand your concern, but I remember a thread where it was 
explained that a RAID0 or linear one basically behaves like a hard drive 
would: since there is no parity and the data is distributed, if say half 
of the devices of the RAID0 are unavailable the LBAs on the other half 
of that RAID will work fine, like if you had a SSD with half of its 
cells broken. So your issue seems to be more related with dealing with 
IO errors than anything. I would imagine that if the filesystem's 
superblock was to become unread/writeable (if it was on the missing 
RAID) then the filesystem would "fail" (be remounted as readonly). Other 
than that there's not much to be done apart from instructing your 
program to stop the IOs and/or fiddling with timeouts to speed up the 
failure of the process.

The "emergency brake" as you put it would work similarly to a RAID5 
losing more than it can: the array will error every write sent to it. 
Alternatively you could disconnect the drives from Linux using the 
"delete" sysfs property. If you use a journalled filesystem you 
shouldn't lose any data over any of this anyways, so that seems safe.

HTH,

Regards,
Ben.