Re: [PATCH] percpu data: only iterate over possible CPUs

[Eric Dumazet <dada1@cosmosbay.com>]

[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]

Andrew Morton a écrit :
> Eric Dumazet <dada1@cosmosbay.com> wrote:
>> Andrew Morton a écrit :
>>> Andi Kleen <ak@muc.de> wrote:
>>>> On Thursday 09 February 2006 19:04, Andrew Morton wrote:
>>>>> Ashok Raj <ashok.raj@intel.com> wrote:
>>>>>> The problem was with ACPI just simply looking at the namespace doesnt
>>>>>>  exactly give us an idea of how many processors are possible in this platform.
>>>>> We need to fix this asap - the performance penalty for HOTPLUG_CPU=y,
>>>>> NR_CPUS=lots will be appreciable.
>>>> What is this performance penalty exactly? 
>>> All those for_each_cpu() loops will hit NR_CPUS cachelines instead of
>>> hweight(cpu_possible_map) cachelines.
>> You mean NR_CPUS bits, mostly all included in a single cacheline, and even in 
>> a single long word :) for most cases (NR_CPUS <= 32 or 64)
>>
> 
> No, I mean cachelines:
> 
> static void recalc_bh_state(void)
> {
> 	int i;
> 	int tot = 0;
> 
> 	if (__get_cpu_var(bh_accounting).ratelimit++ < 4096)
> 		return;
> 	__get_cpu_var(bh_accounting).ratelimit = 0;
> 	for_each_cpu(i)
> 		tot += per_cpu(bh_accounting, i).nr;
> 
> That's going to hit NR_CPUS cachelines even on a 2-way.
> 
> Or am I missing something really obvious here?


OK I see. This can be solved with this patch :

[PATCH] HOTPLUG_CPU : avoid hitting too many cachelines in recalc_bh_state()

Instead of using for_each_cpu(i), we can use for_each_online_cpu(i) : The 
difference matters if HOTPUG_CPU=y

When a CPU goes offline (ie removed from online map), it might have a non null 
bh_accounting.nr, so this patch adds a transfert of this counter to an online 
CPU counter.

We already have a hotcpu_notifier, (function buffer_cpu_notify()), where we 
can do this bh_accounting.nr transfert.

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>

[-- Attachment #2: buffer.patch --]
[-- Type: text/plain, Size: 651 bytes --]

--- a/fs/buffer.c	2006-02-10 15:08:21.000000000 +0100
+++ b/fs/buffer.c	2006-02-10 15:47:55.000000000 +0100
@@ -3138,7 +3138,7 @@
 	if (__get_cpu_var(bh_accounting).ratelimit++ < 4096)
 		return;
 	__get_cpu_var(bh_accounting).ratelimit = 0;
-	for_each_cpu(i)
+	for_each_online_cpu(i)
 		tot += per_cpu(bh_accounting, i).nr;
 	buffer_heads_over_limit = (tot > max_buffer_heads);
 }
@@ -3187,6 +3187,9 @@
 		brelse(b->bhs[i]);
 		b->bhs[i] = NULL;
 	}
+	get_cpu_var(bh_accounting).nr += per_cpu(bh_accounting, cpu).nr ;
+	per_cpu(bh_accounting, cpu).nr = 0;
+	put_cpu_var(bh_accounting);
 }
 
 static int buffer_cpu_notify(struct notifier_block *self,