Re: [RFC]raid5: multiple thread handle stripe

[David Brown <david.brown@hesbynett.no>]

On 16/07/12 06:47, NeilBrown wrote:
> On Mon, 9 Jul 2012 16:00:24 +0800 Shaohua Li<shli@kernel.org>  wrote:
>
>> I had another implementation of raid5 mult-thread. The basic idea is to record
>> stripe submitted CPU. Each cpu has a thread, which only handles stripes
>> submitted from this cpu.
>>
>> Dan mentioned similar idea several day ago. I used to think we need make
>> conf->device_lock per-cpu to make this work, but turn out that isn't required.
>> Simply using percpu list and still using global device_lock works here.
>> Performance is good too.
>>
>> This is a RFC patch, I'll resubmit in a better reviewable way if you like the idea.
>
> I mostly like this, thanks.  It seems to take a fairly simple approach -
> having a single list of pending stripes and each thread just takes the ones
> that are relevant for it.
>
> I'm not convinced of the idea of having one thread for every CPU though. I
> wonder if we should be more NUMA-aware.
> E.g. on an Intel core with SMT we want at most one thread per core, not one
> thread per control thread.
> Similarly in a larger NUMA machine we might want one or 2 threads per node
> even if they have multiple CPUs.  Does that seem reasonable?
>

Raid can't be the only task in the kernel that has this sort of balanced 
requirement of multiple threads, but for which one thread per virtual 
cpu would be overkill.  A brief look at my own system shows multiple 
per-cpu threads for things like block devices and ext4 systems, which 
would surely have the same requirements here.  It strikes me that if it 
is currently not considered excessive to have one thread per cpu per 
mounted ext4 filesystem, for example, then one thread per cpu per raid 
device is not going to have excessive overheads either.  Conversely, if 
it /does/ make sense to limit the thread count for raid on larger 
machines, then perhaps the same logic should be used for other related 
tasks such as block drivers and filesystem threads.

mvh.,

David