RE: [patch 0/4] ia64 SPARSEMEM

Dave Hansen
2005-05-31T21:58:22Z

On Tue, 2005-05-31 at 14:41 -0700, Luck, Tony wrote:
> >* It has good tlb behavior.
> True ... definitely better than VIRTUAL_MEM_MAP. But what effect does
> this have on system level performance?

It slightly improves performance on everything I've run it on, at least
compared to discontigmem. That means a few ppc64 configurations, x86
summit, and NUMAQ.

> >* It is faster and has a lower icache footprint than existing
> > discontigmem implementations.
> Did I miss some benchmark results?

I've posted them a few times. The gain is somewhere in the 1-2% on
NUMAQ. Nothing substantial.

I can dig the results up again, but they're going to mean close to
nothing on your hardware. I'd suggest running it yourself, and seeing
exactly how it behaves.

> >* On a theoretical 16TB ppc64 system with 16MB sections, the overhead of
> > the mem_section[] table is 8MB.
> Back to the "somewhat sparse" arguments of point #1. In fact this theoretical
> system isn't "sparse" at all!

Well, the overhead is still 8MB, even if the system only has 32MB:
16MB@0 and 16MB@(1TB-16MB). That's pretty sparse.

In any case, I agree that the current code isn't optimal across all ia64
platforms. But, I don't think we're seriously tied to that single, flat
array. It's just the easiest way to do it for now.

> >Also, nothing seriously confines us to a flat array of mem_sections,
> >that's just the only implementation right now. The pagetables that are
> >walked in the TLB miss handler (for vmem_map[]) could just as easily be
> >a set of two-level mem_section tables that are walked in software. That
> >just adds an extra load to the pfn_to_page() path. Plus, if somebody
> ^^^^^^^^^^^^^^^^^^^^^^^
> >does this, all sparsemem architectures can benefit.
>
> What would the performance impacts of this extra load be? pfn_to_page()
> appears to be a pretty common operation.

On a normal, x86 flatmem system there's a single load to do
pfn_to_page() from *mem_map. With today's sparsemem, that goes to two
loads (page->flags and mem_section[section]). I haven't been able to
measure the effect of this extra load on any macro-benchmarks.

-- Dave