Re: [RFC 2.6.11-rc2-mm2 0/7] mm: manual page migration -- overview II

[Ray Bryant <raybry@sgi.com>]

All,

Just an update on the idea of migrating a process without suspending
it.

The hard part of the problem here is to make sure that the page_migrate()
system call sees all of the pages to migrate.  If the process that is
being migrated can still allocate pages, then the page_migrate() call
may miss some of the pages.

One way to solve this problem is to force the process to start allocating
pages on the new nodes before calling page_migrate().  There are a couple
of subcases:

(1)  For memory mapped files with a non-DEFAULT associated memory policy,
      one can use mbind() to fixup the memory policy.  (This assumes the
      Steve Longerbeam patches are applied, as I understand things).

(2)  For anonymous pages and memory mapped files with DEFAULT policy,
      the allocation depends on which node the process is running.  So
      after doing the above, you need to migrate the task to a cpu
      associated with one of the nodes.

The problem with (1) is that it is racy, there is no guarenteed way to get the
list of mapped files for the process while it is still running.  A process
can do it for itself, so one way to do this would be to write the set of
new nodes to a /proc/pid file, then send the process a SIG_MIGRATE
signal.  Ugly....  (For multithreaded programs, all of the threads have
to be signalled to keep them from mmap()ing new files during the migration.)

(1) could be handled as part of the page_migrate() system call --
make one pass through the address space searching for mempolicy()
data structures, and updating them as necessary.  Then make a second
pass through and do the migrations.  Any new allocations will then
be done under the new mempolicy, so they won't be missed.  But this
still gets us into trouble if the old and new node lists are not
disjoint.

This doesn't handle anonymous memory or mapped files associated with
the DEFAULT policy.  A way around that would be to add a target cpu_id
to the page_migrate() system call.  Then before doing the first pass
described above, one would do the equivalenet of set_sched_affinity()
for the target pid, moving it to the indicated cpu.  Once it is known
the pid has moved (how to do that?), we now know anonymous memory and
DEFAULT mempolicy mapped files will be allocated on the nodes associated
with the new cpu.  Then we can proceed as discussed in the last paragraph.
Also ugly, due to the extra parameter.

Alternatively, we can just require, for correct execution, the invoking
code to do the set_sched_affinity() first, in those cases where
migrating a running task is important.

Anyway, how important is this, really for acceptance of a page_migrate()
system call in the community?  (that is, how important is it to be
able to migrate a process without suspending it?)
-- 
Best Regards,
Ray
-----------------------------------------------
                   Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
raybry@sgi.com             raybry@austin.rr.com
The box said: "Requires Windows 98 or better",
            so I installed Linux.
-----------------------------------------------