RCU fanout leaf balancing

RCU fanout leaf balancing

[Daniel J Blueman]

Hi Paul,

Finding earlier reference to increasing RCU fanout leaf for the purpose 
of "decrease[ing] cache-miss overhead for large systems", would your 
suggestion be to increase the value to the next hierarchy core-count 
above 16?

If we have say 32 interconnected 48-core servers; 3 sockets of dual-node 
8-core Opteron 6300s, so 1536 cores in all. Latency across the coherent 
interconnect is O(100x) higher than the internal Hypertransport 
interconnect, so if we set RCU_FANOUT_LEAF to 48 to keep leaf-checking 
local to one Hypertransport fabric, what wisdom would one use for 
RCU_FANOUT? 4x leaf?

Or, would it be more cache-friendly to set RCU_FANOUT_LEAF to 8 and 
RCU_FANOUT to 48?

Many thanks,
   Daniel
-- 
Daniel J Blueman
Principal Software Engineer, Numascale

Re: RCU fanout leaf balancing

[Paul E. McKenney]

On Sat, Oct 25, 2014 at 02:59:24PM +0800, Daniel J Blueman wrote:
> Hi Paul,
> 
> Finding earlier reference to increasing RCU fanout leaf for the
> purpose of "decrease[ing] cache-miss overhead for large systems",
> would your suggestion be to increase the value to the next hierarchy
> core-count above 16?
> 
> If we have say 32 interconnected 48-core servers; 3 sockets of
> dual-node 8-core Opteron 6300s, so 1536 cores in all. Latency across
> the coherent interconnect is O(100x) higher than the internal
> Hypertransport interconnect, so if we set RCU_FANOUT_LEAF to 48 to
> keep leaf-checking local to one Hypertransport fabric, what wisdom
> would one use for RCU_FANOUT? 4x leaf?
> 
> Or, would it be more cache-friendly to set RCU_FANOUT_LEAF to 8 and
> RCU_FANOUT to 48?

The easiest approach would be to use the default of 16.  Assuming
consecutive CPU numbering within each 48-core server, this would mean that
you would have three rcu_node structures per 48-core server.  The next
level up would of course span servers, but that level is accessed much
less frequently than is the root level, so this should still work.

If you also have hyperthreading, so that there are 96 hardware threads
per server, and if you are using the same "interesting" numbering scheme
that Intel uses, then this still works.  You have three leaf rcu_node
structure for the first set of hardware threads and another set of three
for the second set of hardware threads.

Or are you seeing some problem with the default?  If so, please tell me
what that problem is.

You can of course increase RCU_FANOUT to 24 or 48 (this latter assuming
a 64-bit kernel), at least if you are using a recent kernel.  However,
the penalty for too large a value for RCU_FANOUT is lock contention at
scheduling-clock-interrupt time.  So if you are setting RCU_FANOUT to 48,
you probably also want to boot with skew_tick set.

But the best approach is to try it.  I bet that the default will work
just fine for you.  ;-)

							Thanx, Paul

Re: RCU fanout leaf balancing

[Daniel J Blueman]

On 25/10/2014 21:48, Paul E. McKenney wrote:
> On Sat, Oct 25, 2014 at 02:59:24PM +0800, Daniel J Blueman wrote:
>> Hi Paul,
>>
>> Finding earlier reference to increasing RCU fanout leaf for the
>> purpose of "decrease[ing] cache-miss overhead for large systems",
>> would your suggestion be to increase the value to the next hierarchy
>> core-count above 16?
>>
>> If we have say 32 interconnected 48-core servers; 3 sockets of
>> dual-node 8-core Opteron 6300s, so 1536 cores in all. Latency across
>> the coherent interconnect is O(100x) higher than the internal
>> Hypertransport interconnect, so if we set RCU_FANOUT_LEAF to 48 to
>> keep leaf-checking local to one Hypertransport fabric, what wisdom
>> would one use for RCU_FANOUT? 4x leaf?
>>
>> Or, would it be more cache-friendly to set RCU_FANOUT_LEAF to 8 and
>> RCU_FANOUT to 48?
>
> The easiest approach would be to use the default of 16.  Assuming
> consecutive CPU numbering within each 48-core server, this would mean that
> you would have three rcu_node structures per 48-core server.  The next
> level up would of course span servers, but that level is accessed much
> less frequently than is the root level, so this should still work.
>
> If you also have hyperthreading, so that there are 96 hardware threads
> per server, and if you are using the same "interesting" numbering scheme
> that Intel uses, then this still works.  You have three leaf rcu_node
> structure for the first set of hardware threads and another set of three
> for the second set of hardware threads.
>
> Or are you seeing some problem with the default?  If so, please tell me
> what that problem is.
>
> You can of course increase RCU_FANOUT to 24 or 48 (this latter assuming
> a 64-bit kernel), at least if you are using a recent kernel.  However,
> the penalty for too large a value for RCU_FANOUT is lock contention at
> scheduling-clock-interrupt time.  So if you are setting RCU_FANOUT to 48,
> you probably also want to boot with skew_tick set.
>
> But the best approach is to try it.  I bet that the default will work
> just fine for you.  ;-)

Good info. I'll stick with the defaults of 16/64, will schedule some 
tuning later, and let you know if I find anything significant.

Thanks Paul!

Daniel
-- 
Daniel J Blueman
Principal Software Engineer, Numascale