Re: md RAID with enterprise-class SATA or SAS drives

[David Brown <david.brown@hesbynett.no>]

On 22/05/2012 08:36, Stan Hoeppner wrote:
> On 5/21/2012 6:34 PM, NeilBrown wrote:
>> On Mon, 21 May 2012 13:51:21 -0500 Stan Hoeppner<stan@hardwarefreak.com>
>> wrote:
>>
>>> On 5/21/2012 10:20 AM, CoolCold wrote:
>>>> On Sat, May 12, 2012 at 2:28 AM, Stan Hoeppner<stan@hardwarefreak.com>  wrote:
>>>>> On 5/11/2012 3:16 AM, Daniel Pocock wrote:
>>>>>
>>>> [snip]
>>>>> That's the one scenario where I abhor using md raid, as I mentioned.  At
>>>>> least, a boot raid 1 pair.  Using layered md raid 1 + 0, or 1 + linear
>>>>> is a great solution for many workloads.  Ask me why I say raid 1 + 0
>>>>> instead of raid 10.
>>>> So, I'm asking - why?
>>>
>>> Neil pointed out quite some time ago that the md RAID 1/5/6/10 code runs
>>> as a single kernel thread.  Thus when running heavy IO workloads across
>>> many rust disks or a few SSDs, the md thread becomes CPU bound, as it
>>> can only execute on a single core, just as with any other single thread.
>>
>> This is not the complete truth.
>
> Yes, I should have stipulated only writes are limited to a single thread.
>
>> For RAID1 and RAID10, successful IO requests do not involved the kernel
>> thread, so the fact that there is only one should be irrelevant.
>> Failed requests are retried using the thread and it is also involved it
>> resync/recovery so those processes may be limited by the single thread.
>>
>> RAID5/6 does not use the thread for read requests on a non-degraded array.
>> However all write requests go through the single thread so there could be
>> issues there.
>
> Thanks for clarifying this.  In your previous response to this issue
> (quoted and linked below) you included RAID 1/10 with RAID 5/6 WRT
> writes going through a single thread.
>
>> Have you  actually measured md/raid10 being slower than raid0 over raid1?
>
> I personally have not, as I don't have access to the storage hardware
> necessary to sink a sufficiently large write stream to peak a core with
> the md thread.
>
>> I have a vague memory from when this came up before that there was some extra
>> issue that I was missing, but I cannot recall it just now....
>
> We're recalling the same thread, which was many months ago.  Here's your
> post:  http://marc.info/?l=linux-raid&m=132616899005148&w=2
>
> And here's the relevant section upon which I was basing my recent
> statements:
>
> "I think you must be misremembering.  Neither RAID0 or Linear have any
> threads involved.  They just redirect the request to the appropriate
> devices.  Multiple threads can submit multiple requests down through
> RAID0 and Linear concurrently.
>
> RAID1, RAID10, and RAID5/6 are different.  For reads they normally are
> have no contention with other requests, but for writes things do get
> single-threaded at some point."
>

I would think that even if writes to raid1 and raid10 do go through a 
single thread, it is unlikely to be a bottleneck - after all, it will 
mostly just pass the write on to the block layer for the 2 (or more) disks.

As for how much single-threading limits raid5/6 writes, it comes down to 
a balance between memory bandwidth and processor speed.  I would imagine 
that for calculating the simple XOR for raid5, the limit is how fast the 
data can get on and off the chip, rather than how fast a single thread 
can chew through it.  If that's the case, then having two threads doing 
the same thing on different blocks will not run any faster.  If you have 
more than one chip, however, you might have more memory bandwidth - and 
raid6 calculations as well as degraded array access involve more 
processing, and will then be cpu limited.

And if the single thread has to block (such as while waiting for reads 
during a partial stripe update on raid5 or raid6), then it could quickly 
become a bottleneck.

But in general, it's important to do some real-world testing to 
establish whether or not there really is a bottleneck here.  It is 
counter-productive for Stan (or anyone else) to advise against raid10 or 
raid5/6 because of a single-thread bottleneck if it doesn't actually 
slow things down in practice.  On the other hand, if it /is/ a hinder to 
scaling, then it is important for Neil and other experts to think about 
how to change the architecture of md raid to scale better.  And 
somewhere in between there can be guidelines to help users - something 
like "for an average server, single-threading will saturate raid5 
performance at 8 disks, raid6 performance at 6 disks, and raid10 at 10 
disks, beyond which you should use raid0 or linear striping over two or 
more arrays".

Of course, to do such testing, someone would need a big machine with 
lots of disks, which is not otherwise in use!

mvh.,

David