Re: RFC/POC Make Page Tables Relocatable

[mel@skynet.ie (Mel Gorman)]

On (25/10/07 11:08), Dave Hansen didst pronounce:
> On Thu, 2007-10-25 at 13:40 -0400, Ross Biro wrote: 
> > On 10/25/07, Dave Hansen <haveblue@us.ibm.com> wrote:
> > > On Thu, 2007-10-25 at 11:16 -0400, Ross Biro wrote:
> > > > 1) Add a separate meta-data allocation to the slab and slub allocator
> > > > and allocate full pages through kmem_cache_alloc instead of get_page.
> > > > The primary motivation of this is that we could shrink struct page by
> > > > using kmem_cache_alloc to allocate whole pages and put the supported
> > > > data in the meta_data area instead of struct page.
> > >
> > > The idea seems cool, but I think I'm missing a lot of your motivation
> > > here.
> > >
> > > First of all, which meta-data, exactly, is causing 'struct page' to be
> > > larger than it could be?  Which meta-data can be moved?
> > 
> > Almost all of it.  Most of struct page isn't about the kernel manging
> > pages in general, but about managing particular types of pages.
> > Although it's been cleaned up over the years, there are still
> > some things:
> > 
> >         union {
> >                 atomic_t _mapcount;     /* Count of ptes mapped in mms,
> >                                          * to show when page is mapped
> >                                          * & limit reverse map searches.
> >                                          */
> >                 struct {        /* SLUB uses */
> >                         short unsigned int inuse;
> >                         short unsigned int offset;
> >                 };
> >         };
> > 
> > mapcount is only used when the page is mapped via a pte, while the
> > other part is only used when the page is part of a SLUB cache.
> > Neither of which is always true and not 100% needed as part of struct
> > page.  There is just currently no better place to put them.  The rest
> > of the unions don't really belong in struct page.  Similarly the lru
> > list only applies to pages which could go on the lru list.  So why not
> > make a better place to put them.
> 
> Right, but we're talking about pagetable pages here, right?  What fields
> in 'struct page' are used by pagetable pages, but will allow 'struct
> page' to shrink in size if pagetables pages stop using them?
> 
> On a more general note: so it's all about saving memory in the end?
> Making 'struct page' smaller?  If I were you, I'd be very conerned about
> the pathological cases.  We may get the lru pointers out of 'struct
> page', so we'll need some parallel lookup to get from physical page to
> LRU, right?   Although the bootup footprint of mem_map[] (and friends)
> smaller, what happens on a machine with virtually all its memory used by
> pages on the LRU (which I would guess is actually quite common).  Will
> the memory footprint even be close to the two pointers per physical page
> that it cost us for the current implementation?
> 
> That doesn't even consider the runtime overhead of such a scheme.  Right
> now, if you touch any part of 'struct page' on a 32-bit machine, you
> generally bring the entire thing into a single cacheline.  Every other
> subsequent access is essentially free.  Any ideas what the ballpark
> number of cachelines are that would have to be brought in with another
> lookup method for 'struct page' to lru?
> 
> I dunno.  I'm highly skeptical this can work.
> 
> I've heard rumors in the past that the Windows' 'struct page' is much
> smaller than the Linux one.  But, I've also heard that this weighs
> heavily in other areas such as page reclamation.  Could be _completely_
> bogus, but it might be worth a search or two to see if there have been
> any papers on the subject.  
> 
> > > get a pte page back, I might simply hold the page table lock, walk the
> > > pagetables to the pmd, lock and invalidate the pmd, copy the pagetable
> > > contents into a new page, update the pmd, and be on my merry way.  Why
> > > doesn't this work?  I'm just fishing for a good explanation why we need
> > > all the slab silliness.
> > 
> > This would almost work, but to do it properly, you find you'll need
> > some more locks and a couple of extra pointers and such.
> 
> Could you be specific?
> 
> > With out all
> > the slab silliness you would need to add them to struct page. It would
> > have needlessly bloated struct page hence the previous change.  I've
> > also managed to convince myself that using the slab/slub allocator
> > will tend to clump the page tables together which should reduce
> > fragmentation and make more memory available for huge pages.  In fact,
> > I've got this idea that by using slab/slub, we can stop allocating
> > individual pages and only allocate huge pages on systems that have
> > them.
> 
> You may want to have a talk with Mel about memory fragmentation, and
> whether there is any lower hanging fruit (cc'd). :)
> 

I suspect this might be overkill from a memory fragmentation
perspective. When grouping pages by mobility, page table pages are
currently considered MIGRATE_UNMOVABLE. From what I have seen, they are
by far the most common unmovable allocation. If they were relocatable
with the standard page migration mechanism, they could be considered
MIGRATE_MOVABLE and external fragmentation would be easier to content
with.

I haven't looked closely enough at this patch to know if moving page
table pages with page migration is the aim or not.

However, using huge pages just all slabs does not feel like a great
idea. There will be a lot memory wasted due to internal fragmentation
and systems with less memory are not going to want to commit a hugepage
for a small slab allocation.

> > > You might also want to run checkpatch.pl on your patch.  It has some
> > > style issues that also need to get worked out.
> > 
> > That patch isn't meant to be applied, but is there because it's easier
> > to point to code to try to explain what I'm mean than to explain in
> > words.  I didn't think a few style issues would be an issue.  And just
> > to reiterate, if you actually use the code I posted, you get what you
> > deserve.  It was only meant to illustrate what I'm trying to say.
> 
> In general, the reason to run such a script (and to have coding
> standards in the first place) is so that others can more easily read
> your code.  The posted patch is hard to understand in some areas because
> of indenting bracketing.  If you'd like people to read, review, and give
> suggestions on what they see, I'd suggest trying to make it as easy as
> possible to understand.
> 
> Check out Documentation/CodingStyle.  
> 
> -- Dave
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>