[PATCH] x86_64: Double the per cpu area size

[PATCH] x86_64: Double the per cpu area size

[Eric W. Biederman]

With the support for more than 256 irqs kernel_stat.irqs has
grown noticeably in size.  When asked about it no one wants to
keep less information, so we need a larger per cpu area.

According to my rough calculations the per cpu usage
has grown by about 15K (in the worst case) while the per cpu
area size has grown by 32K so we should have enough room for
a while.

Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
 include/asm-x86_64/percpu.h |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/include/asm-x86_64/percpu.h b/include/asm-x86_64/percpu.h
index 08dd9f9..56e3bdb 100644
--- a/include/asm-x86_64/percpu.h
+++ b/include/asm-x86_64/percpu.h
@@ -11,6 +11,8 @@ #ifdef CONFIG_SMP
 
 #include <asm/pda.h>
 
+#define PERCPU_ENOUGH_ROOM 65536
+
 #define __per_cpu_offset(cpu) (cpu_pda(cpu)->data_offset)
 #define __my_cpu_offset() read_pda(data_offset)
 
-- 
1.4.2.rc3.g7e18e

Re: [PATCH] x86_64: Double the per cpu area size

[Andi Kleen]

>  
>  #include <asm/pda.h>
>  
> +#define PERCPU_ENOUGH_ROOM 65536

I would prefer if you didn't do that unconditionally. Can you make
it dependent on NR_IRQS or so?  Can you add a test for CONFIG_TINY
to make it smaller?

Also longer term it should really properly fixed

-Andi

Re: [PATCH] x86_64: Double the per cpu area size

[Eric W. Biederman]

Andi Kleen <ak@suse.de> writes:

>>  
>>  #include <asm/pda.h>
>>  
>> +#define PERCPU_ENOUGH_ROOM 65536
>
> I would prefer if you didn't do that unconditionally. Can you make
> it dependent on NR_IRQS or so?  Can you add a test for CONFIG_TINY
> to make it smaller?

We already ignore this variable for the per cpu allocation if we
build a kernel wihtout module support.  I guess a good fix could
entail changing the concept to be how much per cpu room to reserve
for modules.

I'm a big believer in stupid and simple solutions for as long as
you can get away with it.  When people trip over this then 
it will be clear what the real problem is and we can fix it.

> Also longer term it should really properly fixed

Agreed.  But we need a solution that works now so we have a solution
for when the 2.6.19 window opens up.  There is no agreement on even
what a proper fix is, or even what it looks like.  Keeping the data
per cpu seems about as good as anything else for memory size savings,
as most systems don't have many cpus.

Throwing a few more bytes at the problem solves it today and for
all systems currently built.  This buys us time to look at and discuss
the problem.  With MSI starting to be useful I have no expectation
that we will stop here.

There are two fundamental problems that need to be fixed.
- Small size and static allocate of the per cpu area.
- Data structures that don't scale to large numbers of possible irqs.

Solving either of these two fundamental problems involves reexamining
some of our current trade offs in the kernel.

The proper fix for irqs is a refactoring of the data structures so
we can handle a 16 or better yet a 24 bit irq number, and only
allocate the pieces we need.  A proper fix needs to find someway
not to keep a counter for every cpu and irq pair, which no one has
the will to seriously consider right now.

The proper fix for the per cpu area size is much trickier.
Especially if we every reach the point of hotplug NUMA nodes.
One odd observation is that the amount of per cpu data we want
grows with the size of the system.  

Eric

Re: [PATCH] x86_64: Double the per cpu area size

[Keith Mannthey]

On 8/7/06, Eric W. Biederman <ebiederm@xmission.com> wrote:
> Andi Kleen <ak@suse.de> writes:
>
> >>
> >>  #include <asm/pda.h>
> >>
> >> +#define PERCPU_ENOUGH_ROOM 65536

Is it possible to put this into -mm untill things are sorted?
2.6.18-rc3-mm2 x86_64 is still without any per-cpu space for modules.

Thanks,
  Keith

Re: [PATCH] x86_64: Double the per cpu area size

[Eric W. Biederman]

"Keith Mannthey" <kmannth@gmail.com> writes:

> On 8/7/06, Eric W. Biederman <ebiederm@xmission.com> wrote:
>> Andi Kleen <ak@suse.de> writes:
>>
>> >>
>> >>  #include <asm/pda.h>
>> >>
>> >> +#define PERCPU_ENOUGH_ROOM 65536
>
> Is it possible to put this into -mm untill things are sorted?
> 2.6.18-rc3-mm2 x86_64 is still without any per-cpu space for modules.

I think we are sorted (see the later patch to auto size the per cpu
area).  But you should be ok with current -mm if you compile for 64 or
fewer cpus.

Eric

Re: [PATCH] x86_64: Double the per cpu area size

[Keith Mannthey]

On 8/8/06, Eric W. Biederman <ebiederm@xmission.com> wrote:
> "Keith Mannthey" <kmannth@gmail.com> writes:
>
> > On 8/7/06, Eric W. Biederman <ebiederm@xmission.com> wrote:
> >> Andi Kleen <ak@suse.de> writes:
> >>
> >> >>
> >> >>  #include <asm/pda.h>
> >> >>
> >> >> +#define PERCPU_ENOUGH_ROOM 65536
> >
> > Is it possible to put this into -mm untill things are sorted?
> > 2.6.18-rc3-mm2 x86_64 is still without any per-cpu space for modules.
>
> I think we are sorted (see the later patch to auto size the per cpu
> area).  But you should be ok with current -mm if you compile for 64 or
> fewer cpus.

Yep it looks like the config file I tote around on this box has
CONFIG_NR_CPUS=128. I thought this this was the default NR_CPUS but it
is not.

Thanks,
  Keith