Re: Handling NUMA page migration

[Frank Mehnert <frank.mehnert@oracle.com>]

[-- Attachment #1: Type: Text/Plain, Size: 3028 bytes --]

On Wednesday 05 June 2013 12:10:19 Mel Gorman wrote:
> On Tue, Jun 04, 2013 at 06:58:07AM -0500, Robin Holt wrote:
> > > B) 1. allocate memory with alloc_pages()
> > > 
> > >    2. SetPageReserved()
> > >    3. vm_mmap() to allocate a userspace mapping
> > >    4. vm_insert_page()
> > >    5. vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP)
> > >    
> > >       (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND |
> > >       0xff)
> > > 
> > > At least the memory allocated like B) is affected by automatic NUMA
> > > page migration. I'm not sure about A).
> > > 
> > > 1. How can I prevent automatic NUMA page migration on this memory?
> > > 2. Can NUMA page migration also be handled on such kind of memory
> > > without
> > > 
> > >    preventing migration?
> 
> Page migration does not expect a PageReserved && PageLRU page. The only
> reserved check that is made by migration is for the zero page and that
> happens in the syscall path for move_pages() which is not used by either
> compaction or automatic balancing.
> 
> At some point you must have a driver that is setting PageReserved on
> anonymous pages that is later encountered by automatic numa balancing
> during a NUMA hinting fault.  I expect this is an out-of-tree driver or
> a custom kernel of some sort. Memory should be pinned by elevating the
> reference count of the page, not setting PageReserved.

Yes, this is ring 0 code from VirtualBox. The VBox ring 0 driver does the
steps which are shown above. Setting PageReserved is not only for pinning
but also for fork() protection. I've tried to do get_page() as well but
it did not help preventing the migration during NUMA balancing.

As I wrote, the code for allocating + mapping the memory assumes that
the memory is finally pinned and will be never unmapped. That assumption
might be wrong or wrong under certain/rare conditions. I would like to
know these conditions and how we can prevent them from happening or how
we can handle them correctly.

> It's not particularly clear how you avoid hitting the same bug due to THP
> and memory compaction to be honest but maybe your setup hits a steady
> state that simply never hit the problem or it happens rarely and it was
> not identified.

I'm currently using the stock Ubuntu 13.04 generic kernel (3.8.0-23),
patched with some additional logging code. It is true that this problem
could also be triggered by other kernel mechanisms as you described.

Thanks,

Frank
-- 
Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox
ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany

Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Geschäftsführer: Jürgen Kunz

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]