Re: sparsemem support for mips with highmem

[Christoph Lameter <cl@linux-foundation.org>]

David VomLehn wrote:
> 
> For a flat memory model, the page descriptors array memmap is
> contiguously allocated in low memory. For sparse memory, you only
> allocate memory to hold page descriptors that actually exist. If you
> don't enable CONFIG_SPARSEMEM_VMEMMAP, you introduce a level of
> indirection where the top bits of an address gives you an index into an
> array that points to an array of page descriptors for that section of
> memory. This has some performance impact relative to flat memory due to
> the extra memory access to read the pointer to the array of page
> descriptors.

Right.

> If you do enable CONFIG_SPARSEMEM_VMEMMAP, you still allocate memory to
> hold page descriptors, but you map that memory into virtual space so
> that a given page descriptor for a physical address is at the offset
> from the beginning of the virtual memmap corresponding to the page frame
> number of that address. This gives you a single memmap, just like you
> had in the flat memory case, though memmap now lives in virtual address
> space. Since memmap now lives in virtual address space, you don't need
> to use any memory to back the virtual addresses that correspond to the
> holes in your physical memory, which is how you save a lot of physical
> memory. The performance impact relative to flag memory is now that of
> having to go through the TLB to get to the page descriptor.

Correct.

> If you are using CONFIG_SPARSEMEM_VMEMMAP and the corresponding TLB
> entry is present, you expect this will be faster than the extra memory
> access you do when CONFIG_SPARSEMEM_VMEMMAP is not enabled, even if that
> memory is in cache. This seems like a pretty reasonable expectation to
> me. Since TLB entries cover much more memory than the cache, it also
> seems like there would be a much better chance that you already have the
> corresponding TLB entry than having the indirect memory pointer in
> cache. And, in the worst case, reading the TLB entry is just another
> memory access, so it's closely equivalent to not enabling
> CONFIG_SPARSEMEM_VMEMMAP.

Exactly.

> So, if I understand this right, the overhead on a MIPS processor using
> flat memory versus using sparse memory with CONFIG_SPARSEMEM_VMEMMAP
> enabled would be mostly the difference between accessing unmapped
> memory, which doesn't go through the TLB, and mapped memory, which does.
> Even though there is some impact due to TLB misses, this should be
> pretty reasonable. What a way cool approach!

Great. Thanks.