Re: Page allocator: Single Zone optimizations

Christoph Lameter
2006-11-03T21:42:16Z

On Fri, 3 Nov 2006, Mel Gorman wrote:

> > I think we need something like what is done here via anti-frag but I wish
> > it would be more generic and not solely rely on reclaim to get pages freed
> > up.
> >
>
> How could it have been made more generic? Fundamentally, all we are doing at
> the moment is using the freelists to cluster types of pages together. We only
> depend on reclaim now. If we get the clustering part done, I can start working
> on the page migration part.

Right lets have a special freelist for unreclaim/unmovable pages. I think
that we agree on m that. Somehow we need to be able to insure that
unXXXable pages do not end up in sections of the zone where we allow
memory hotplug.

At some later point we would like to have the ability to redirect
unXXXable allocations to another node if the node is hot pluggable.

> > Also the duplication of the page struct caches worries me because it
> > reduces the hit rate.
>
> do you mean the per-cpu caches? If so, without clustering in the per-cpu
> caches, unmovable allocations would "leak" into blocks used for movable
> allocations.

I mean the per cpu caches and I think you could just bypass the per cpu
caches for unXXXable pages. Kernel pages are buffered already in the slab
allocator and other kernel allocations are probably rare enough.

> > Removing the intermediate type would reduce the page
> > caches to 2.
>
> And significantly reduce the effectiveness of the clustering in the process.

Are you sure about this? It seems that the intermediate type is
reclaimable and you already allow "reclaimable" pages to be not reclaimable
(mlock'ed pages). If you run into trouble with the reclaimable slab pages
in the reclaimable zone then you could do agressive slab reclaim to remedy
the situation.

> > And maybe we do not need caches for unreclaimable/unmovable
> > pages? slab already does its own buffering there.
> That is true. If it is a problem, what could be done is have a per-cpu cache
> for movable and unmovable allocations. Then have the __GFP_KERNRCLM
> allocations bypass the per-cpu allocator altogether and go straight to the
> buddy allocator.

Right. Maybe we can get away with leaving the pageset cpu caches
untouched? On our largest systems with 1k nodes 4k cpus we currently have
4 zones * 4096 cpus * 1024 nodes = 16 million pagesets. Each of those has
hot and cold yielding 32 million lists. Now we going triplicate that to
192 mio lists and we also increase the size of the structure.

With the code currently in 2.6.19 we go from 4 to 2 zones. So we have only
16 million pagesets. With the optional DMA in mm we got from 16 to 8
million pagesets. This effectively undoes the optimizations done in .19
.20.