Re: regarding the x86_64 zero-based percpu patches

Tejun Heo
2009-01-14T03:59:33Z

Hello, Eric.

Eric W. Biederman wrote:
> Tejun Heo <tj@kernel.org> writes:
>> I don't know. I think it's a dangerous thing which can be avoided.
>> If there's no other solution, then we might have to live with it but I
>> don't see the winning benefit of such design over per-cpu virtual
>> mapping.
>
> It isn't incompatible with a per-cpu virtual mapping. It allows the
> possibility of each cpu reusing the same chunk of virtual address
> space for per cpu memory.
>
> On x86_64 and other architectures with enough address space bits it allows
> us to share the large pages that we use for the normal memory mapping with
> the ones for per cpu access.
>
> I definitely think the work of combining the pda and the percpu areas
> into a common area is worthwhile.

Yeah, it's gonna be necessary regardless of which way we go.

> I think it would be nice if the percpu area could grow and would not be
> a fixed size at boot time, I'm not particularly convinced it has to.

The main problem is that the area needs to be congruent which
basically mandates them to be contiguous. The three alternatives on
table are...

1. Just reserve memory from the get-go. Simplest. No additional TLB
pressure but memory is likely to be wasted and more importantly
scalability suffers.

2. Reserve address space and map memory as necessary. We can be much
more generous about reserving address space especially on 64bit
machines and probably can mostly forget about scalability issue
there. However, getting things just right for address space
contrained 32bit might not be too easy but then again nothing
really is scalable on 32bit these days, so we probably can live
with boot time parameter or something.

Another issue is added TLB pressure as it's likely to consume 4K
TLB entries in addition to the default kernel mapping 2M TLB
entries. The TLB pressure can be mostly avoided if percpu area is
sufficiently large to justify 2MB page allocation but it isn't.

3. Do realloc(). This doesn't impose scalability issues or add to TLB
pressure but it does contrain how the percpu variables can be used
and introduces certain amount of possibility for scary
once-in-a-blue-moon never-reproducible bugs. Maybe such
possibility can be reduced by putting some restriction on the
interface but I don't know. It still scares me.

Hmm... IIUC, the biggest drawback of #2 is the added TLB pressure,
right? What if we reserve percpu allocation by 2MB chunks? ie. use
4k mapping but always allocate the percpu pages from aligned 2MB
chunks. That way it won't waste 2MB per cpu and although it will use
additional 4K TLB entries, it will free up 2MB TLB entries.

Thanks.

--
tejun