Re: [RFC][PATCH] lru_add_drain_all() don't use schedule_on_each_cpu()

[Lee Schermerhorn <Lee.Schermerhorn@hp.com>]

On Tue, 2008-10-28 at 13:45 -0700, Andrew Morton wrote:
> On Tue, 28 Oct 2008 09:25:31 -0500 (CDT)
> Christoph Lameter <cl@linux-foundation.org> wrote:
> 
> > On Mon, 27 Oct 2008, Andrew Morton wrote:
> > 
> > > Can we fix that instead?
> > 
> > How about this fix?
> > 
> > 
> > 
> > Subject: Move migrate_prep out from under mmap_sem
> > 
> > Move the migrate_prep outside the mmap_sem for the following system calls
> > 
> > 1. sys_move_pages
> > 2. sys_migrate_pages
> > 3. sys_mbind()
> > 
> > It really does not matter when we flush the lru. The system is free to add
> > pages onto the lru even during migration which will make the page 
> > migration either skip the page (mbind, migrate_pages) or return a busy 
> > state (move_pages).
> > 
> 
> That looks nicer, thanks.  Hopefully it fixes the
> lockdep-warning/deadlock...

I believe that we still  have the lru_drain_all() called from the fault
path [with mmap_sem held] in clear_page_mlock().  We call
clear_page_mlock() on COW of an mlocked page in a VM_LOCKED vma to
ensure that we don't end up with an mlocked page in some other task's
non-VM_LOCKED vma where we'd then fail to munlock it later.  During
development testing, Rik encountered scenarios where a page would
encounter a COW fault while it was still making its way to the LRU via
the pagevecs.  So, he added the 'drain_all() and that seemed to avoid
this scenario.

Now, in the current upstream version of the unevictable mlocked pages
patches, we just count any mlocked pages [vmstat] that make their way to
free*page() instead of BUGging out, as we were doing earlier during
development.  So, maybe we can drop the lru_drain_add()s in the
unevictable mlocked pages work and live with the occasional freed
mlocked page, or mlocked page on the active/inactive lists to be dealt
with by vmscan.

Comments?

Lee

> 
> I guess we should document our newly discovered schedule_on_each_cpu()
> problems before we forget about it and later rediscover it.