MPOL_INTERLEAVE page-wise binding dump

MPOL_INTERLEAVE page-wise binding dump

[Ananya Muddukrishna]

Hi,

Is there an inexpensive way to determine 
which pages (virtual addresses) 
are bound to which NUMA nodes when 
using MPOL_INTERLEAVE or MPOL_BIND 
with a given set of nodes? 

I want to implement a strict 
round-robin based interleaving 
policy based on MPOL_INTERLEAVE. 

By strict I mean, 
there is no fall back and 
the system call returns an error 
if allocation fails on the given FIFO node set.

I am using numactl-2.0.8-rc3 on linux 2.6.32 
in a large server (4X12-core, 8 NUMA nodes) 
environment.

Please advise. Thank you.
- Ananya

Re: MPOL_INTERLEAVE page-wise binding dump

[Andi Kleen]

On Thu, Dec 22, 2011 at 05:12:41PM +0000, Ananya Muddukrishna wrote:
> Hi,
> 
> Is there an inexpensive way to determine 
> which pages (virtual addresses) 
> are bound to which NUMA nodes when 
> using MPOL_INTERLEAVE or MPOL_BIND 
> with a given set of nodes? 


get_mempolicy MPOL_F_ADDR|MPOL_F_NODE

> 
> I want to implement a strict 
> round-robin based interleaving 
> policy based on MPOL_INTERLEAVE. 

What's the rationale?


-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: MPOL_INTERLEAVE page-wise binding dump

[Ananya Muddukrishna]

Hi,

> On Thu, 22 Dec 2011 20:58:08 +0100
> Andi Kleen <andi@firstfloor.org> wrote:

> On Thu, Dec 22, 2011 at 05:12:41PM +0000, Ananya Muddukrishna wrote:
> > Hi,
> > 
> > Is there an inexpensive way to determine 
> > which pages (virtual addresses) 
> > are bound to which NUMA nodes when 
> > using MPOL_INTERLEAVE or MPOL_BIND 
> > with a given set of nodes? 
> 
> 
> get_mempolicy MPOL_F_ADDR|MPOL_F_NODE
> 

Thanks! I will check this out and get back. I hope this returns the
affinity as a page-to-node map.

> > 
> > I want to implement a strict 
> > round-robin based interleaving 
> > policy based on MPOL_INTERLEAVE. 
> 
> What's the rationale?

The rationale relates to scheduling tasks (OpenMP and the like) with
page affinity in a user-level runtime system (work-crew threading
model).

The runtime system allocates memory pages in a deterministic way using
mbind and from then on, schedules tasks such that accesses to task
operated pages (affinity) do not incur large NUMA overheads. I am
thinking of using mlock so that swapin_readahead does not affect the
affinity of runtime-allocated pages. The first-touch policy is not of
much help since the allocated memory is initialized before tasks are
scheduled. 

> 
> 
> -Andi
> -- 
> ak@linux.intel.com -- Speaking for myself only.

-- Ananya 

Re: MPOL_INTERLEAVE page-wise binding dump

[Ananya Muddukrishna]

> > Is there an inexpensive way to determine 
> > which pages (virtual addresses) 
> > are bound to which NUMA nodes when 
> > using MPOL_INTERLEAVE or MPOL_BIND 
> > with a given set of nodes?   
> 
> 
> get_mempolicy MPOL_F_ADDR|MPOL_F_NODE
>   

Thanks! I will check this out and get back. I hope this returns the
affinity as a page-to-node map.

> > 
> > I want to implement a strict 
> > round-robin based interleaving 
> > policy based on MPOL_INTERLEAVE.   
> 
> What's the rationale?  

The rationale relates to scheduling tasks (OpenMP and the like) with
page affinity in a user-level runtime system (work-crew threading
model).

The runtime system allocates memory pages in a deterministic way using
mbind and from then on, schedules tasks such that accesses to task
operated pages (affinity) do not incur large NUMA overheads. I am
thinking of using mlock so that swapin_readahead does not affect the
affinity of runtime-allocated pages. The first-touch policy is not of
much help since the allocated memory is initialized before tasks are
scheduled. 

-- Ananya