Re: [RFC 0/6] mm: support madvise(MADV_FREE)

[Minchan Kim <minchan@kernel.org>]

On Tue, Mar 18, 2014 at 05:23:37PM -0700, Andy Lutomirski wrote:
> On Tue, Mar 18, 2014 at 5:18 PM, Minchan Kim <minchan@kernel.org> wrote:
> > Hello,
> >
> > On Tue, Mar 18, 2014 at 10:55:24AM -0700, Andy Lutomirski wrote:
> >> On 03/13/2014 11:37 PM, Minchan Kim wrote:
> >> > This patch is an attempt to support MADV_FREE for Linux.
> >> >
> >> > Rationale is following as.
> >> >
> >> > Allocators call munmap(2) when user call free(3) if ptr is
> >> > in mmaped area. But munmap isn't cheap because it have to clean up
> >> > all pte entries, unlinking a vma and returns free pages to buddy
> >> > so overhead would be increased linearly by mmaped area's size.
> >> > So they like madvise_dontneed rather than munmap.
> >> >
> >> > "dontneed" holds read-side lock of mmap_sem so other threads
> >> > of the process could go with concurrent page faults so it is
> >> > better than munmap if it's not lack of address space.
> >> > But the problem is that most of allocator reuses that address
> >> > space soonish so applications see page fault, page allocation,
> >> > page zeroing if allocator already called madvise_dontneed
> >> > on the address space.
> >> >
> >> > For avoidng that overheads, other OS have supported MADV_FREE.
> >> > The idea is just mark pages as lazyfree when madvise called
> >> > and purge them if memory pressure happens. Otherwise, VM doesn't
> >> > detach pages on the address space so application could use
> >> > that memory space without above overheads.
> >>
> >> I must be missing something.
> >>
> >> If the application issues MADV_FREE and then writes to the MADV_FREEd
> >> range, the kernel needs to know that the pages are no longer safe to
> >> lazily free.  This would presumably happen via a page fault on write.
> >> For that to happen reliably, the kernel has to write protect the pages
> >> when MADV_FREE is called, which in turn requires flushing the TLBs.
> >
> > It could be done by pte_dirty bit check. Of course, if some architectures
> > don't support it by H/W, pte_mkdirty would make it CoW as you said.
> 
> If the page already has dirty PTEs, then you need to clear the dirty
> bits and flush TLBs so that other CPUs notice that the PTEs are clean,
> I think.

True. I didn't mean we don't need TLB flush. Look at the code although
there are lots of bug in RFC v1.

> 
> Also, this has very odd semantics wrt reading the page after MADV_FREE
> -- is reading the page guaranteed to un-free it?

Yeb, I thought about that oddness but didn't make conclusion because
other OS seem to work like that.
http://www.freebsd.org/cgi/man.cgi?query=madvise&sektion=2

But we could fix it easily by checking access bit instead of dirty bit.

> 
> >>
> >> How does this end up being faster than munmap?
> >
> > MADV_FREE doesn't need to return back the pages into page allocator
> > compared to MADV_DONTNEED and the overhead is not small when I measured
> > that on my machine.(Roughly, MADV_FREE's cost is half of DONTNEED through
> > avoiding involving page allocator.)
> >
> > But I'd like to clarify that it's not MADV_FREE's goal that syscall
> > itself should be faster than MADV_DONTNEED but major goal is to
> > avoid unnecessary page fault + page allocation + page zeroing +
> > garbage swapout.
> 
> This sounds like it might be better solved by trying to make munmap or
> MADV_DONTNEED faster.  Maybe those functions should lazily give pages
> back to the buddy allocator.

About munmap, it needs write-mmap_sem and it hurts heavily of
allocator performance in multi-thread.

About MADV_DONTNEED, Rik van Riel tried to replace MADV_DONTNEED
with MADV_FREE in 2007(http://lwn.net/Articles/230799/).
But I don't know why it was dropped. One think I can imagine
is that it could make regression because user on MADV_DONTNEED
expect rss decreasing when syscall is called.

> 
> --Andy
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>