Re: Migrate pages from a ccNUMA node to another - patch

[Hirokazu Takahashi <taka@valinux.co.jp>]

Hello,

> > I guess aruguments src_node, mm and pte would be redundant since
> > they can be looked up from old_p with the reverse mapping scheme.
> 
> In my version 0.2, I can do with only the following arguments:
>  *		node:	Destination NUMA node
>  *		mm:	-> victim "mm_struct"
>  *		pte:	-> PTE of the page to be moved
> (If I have "mm" at hand, why not to use it ? Why not to avoid fetching the r-map
> page struct ?)
> 
> > >Notes: "pte" can be NULL if I do not know it apriori
> > >       I cannot release "mm->page_table_lock" otherwise I have to re-scan the "mm->pgd".
> > 
> > Re-schan plicy would be much better since migrating pages is heavy work.
> > I don't think that holding mm->page_table_lock for long time would be
> > good idea.
> 
> Re-scanning is "cache killer", at least on IA64 with huge user memory size.
> I have more than 512 Mbytes user memory and its PTEs do not fit into the L2 cache.
> 
> In my current design, I have the outer loops: PGD, PMD and PTE walking; and once
> I find a valid PTE, I check it against the list of max. 2048 physical addresses as
> the inner loop.
> I reversed them: walking through the list of max. 2048 physical addresses as outer
> loop and the PGD - PMD - PTE scans as inner loops resulted in 4 to 5 times slower
> migration.

I've been thinking about it.

I guess our page remap patches would be overkill for your purpose.
Point of our patches is that:
   1. Blocks new access to a specified page.
   2. Waits for the page going into quiescent state.
   3. Copies data from the page to a new page and exchanges them.

In my understanding you want to handle only anonymous pages
which don't have backing store yet. This means that you only need
step 3.

> > How do you think about following algorism:
> >   1. get mm->page_table_lock
> >   2. chose some pages.
> >   3. release mm->page_table_lock
> >   4. call remap_onepage() against each page.
> >   5. goto step1 if there remain pages to be migrated.
> 
> I want to move the most frequently used pages - at least with the HW assisted
> hot page detection.
> I take "mm->page_table_lock", I nuke the PTE. We've got a good chance that the CPU
> using the page observes a page fault almost immediately. It enters the page fault
> handler and gets blocked by "mm->page_table_lock". If I released the lock, the CPU
> could continue and realize that there is nothing to do, the page fault has already
> been repaired. In the mean time, it is me who wait for "mm->page_table_lock".

If you use the HW assisted hot page detection, just notify our remapping
functions of hot pages directly. Everything would be handled well and
pagefault handler would be blocked by PG_lock bit.