Re: [patch] tlb flush_data: replace per_cpu with an array

Peter Zijlstra
2009-01-13T12:00:27Z

On Tue, 2009-01-13 at 00:00 +0100, Ingo Molnar wrote:

> From 23d9dc8bffc759c131b09a48b5215cc2b37a5ac3 Mon Sep 17 00:00:00 2001
> From: Frederik Deweerdt <frederik.deweerdt@xprog.eu>
> Date: Mon, 12 Jan 2009 22:35:42 +0100
> Subject: [PATCH] x86, tlb flush_data: replace per_cpu with an array
>
> Impact: micro-optimization, memory reduction
>
> On x86_64 flush tlb data is stored in per_cpu variables. This is
> unnecessary because only the first NUM_INVALIDATE_TLB_VECTORS entries
> are accessed.
>
> This patch aims at making the code less confusing (there's nothing
> really "per_cpu") by using a plain array. It also would save some memory
> on most distros out there (Ubuntu x86_64 has NR_CPUS=64 by default).
>
> [ Ravikiran G Thirumalai also pointed out that the correct alignment
> is ____cacheline_internodealigned_in_smp, so that there's no
> bouncing on vsmp. ]
>
> Signed-off-by: Frederik Deweerdt <frederik.deweerdt@xprog.eu>
> Signed-off-by: Ingo Molnar <mingo@elte.hu>
> ---
> arch/x86/kernel/tlb_64.c | 16 ++++++++--------
> 1 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/arch/x86/kernel/tlb_64.c b/arch/x86/kernel/tlb_64.c
> index f8be6f1..c5a6c6f 100644
> --- a/arch/x86/kernel/tlb_64.c
> +++ b/arch/x86/kernel/tlb_64.c
> @@ -33,7 +33,7 @@
> * To avoid global state use 8 different call vectors.
> * Each CPU uses a specific vector to trigger flushes on other
> * CPUs. Depending on the received vector the target CPUs look into
> - * the right per cpu variable for the flush data.
> + * the right array slot for the flush data.
> *
> * With more than 8 CPUs they are hashed to the 8 available
> * vectors. The limited global vector space forces us to this right now.
> @@ -48,13 +48,13 @@ union smp_flush_state {
> unsigned long flush_va;
> spinlock_t tlbstate_lock;
> };
> - char pad[SMP_CACHE_BYTES];
> -} ____cacheline_aligned;
> + char pad[X86_INTERNODE_CACHE_BYTES];
> +} ____cacheline_internodealigned_in_smp;

That will make the below array 8*4096 bytes for VSMP, which pushes the
limit for memory savings up to 256 cpus.

I'm really dubious this patch is really worth it.

> /* State is put into the per CPU data section, but padded
> to a full cache line because other CPUs can access it and we don't
> want false sharing in the per cpu data segment. */
> -static DEFINE_PER_CPU(union smp_flush_state, flush_state);
> +static union smp_flush_state flush_state[NUM_INVALIDATE_TLB_VECTORS];
>
> /*
> * We cannot call mmdrop() because we are in interrupt context,
> @@ -129,7 +129,7 @@ asmlinkage void smp_invalidate_interrupt(struct pt_regs *regs)
> * Use that to determine where the sender put the data.
> */
> sender = ~regs->orig_ax - INVALIDATE_TLB_VECTOR_START;
> - f = &per_cpu(flush_state, sender);
> + f = &flush_state[sender];
>
> if (!cpu_isset(cpu, f->flush_cpumask))
> goto out;
> @@ -169,7 +169,7 @@ void native_flush_tlb_others(const cpumask_t *cpumaskp, struct mm_struct *mm,
>
> /* Caller has disabled preemption */
> sender = smp_processor_id() % NUM_INVALIDATE_TLB_VECTORS;
> - f = &per_cpu(flush_state, sender);
> + f = &flush_state[sender];
>
> /*
> * Could avoid this lock when
> @@ -205,8 +205,8 @@ static int __cpuinit init_smp_flush(void)
> {
> int i;
>
> - for_each_possible_cpu(i)
> - spin_lock_init(&per_cpu(flush_state, i).tlbstate_lock);
> + for (i = 0; i < ARRAY_SIZE(flush_state); i++)
> + spin_lock_init(&flush_state[i].tlbstate_lock);
>
> return 0;
> }
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/