Re: [LSF/MM/BPF BoF Session] Numa-Aware Placement for Page Cache Pages

[Ritesh Harjani (IBM) <ritesh.list@gmail.com>]

Gregory Price <gourry@gourry.net> writes:

> On Sat, May 02, 2026 at 03:57:19PM +0100, Gregory Price wrote:
>> On Thu, Apr 30, 2026 at 02:15:19PM +0100, Matthew Wilcox wrote:
>> 
>> fd = open()
>> buf = mmap(fd, ...)
>> mbind(buf, device_node)
>> /* fault file pages directly onto device memory */
>>
>
> maybe more explicitly, something like this
>
> fcntl(fd, F_SET_FILE_NUMA_NODE, gpu_nid); /* pref node */
> mbind(addr, size, MPOL_BIND, ..., MOVE_ALL); /* move existing */


One existing problem with this approach of using MPOL_MF_MOVE[_ALL] (not
specifically about gpu nid usecase here) is that, it only considers
pages which are mapped into process address space. So, unmapped page
cache pages are invisible to it.

So, the problem which I am trying to highlight here is - Today there is
no mechanism for a user to move/migrate the file-cache pages w/o
incurring extra I/Os. The only way that works today is to first populate
all the folios belonging to the file into the process address space by
issuing MADV_POPULATE_READ and then issue mbind(..., MPOL_F_MOVE). But
there is no primitive to populate the process address space with only
the folios which are already present in the page cache (which maybe
residing on a different numa node). Note that MADV_POPULATE_READ will
read the missing pages from disk too.

So, maybe something like this.. Would this still be useful? i.e.

madvise (addr, size, MADV_POPULATE_READ_NOIO);
mbind(addr, size, MPOL_BIND, ..., MPOL_MF_MOVE);

MADV_POPULATE_READ_NOIO should ensure that only the cached folios
belonging to that file are mapped into the process address space w/o
doing any extra disk I/Os. The subsequent mbind call with MPOL_MF_MOVE,
will then ensure that all the existing mapped folios are migrated into the
chosen numa node. And also that any new pages which gets faulted in will
get allocated onto the chose numa node because of MPOL_BIND policy. 

I believe there might be existing applications which might be facing
this problem today. This can happen, for instance, when there is a
workload which can run multiple times and may run across different NUMA
nodes. Our internal test team once reported a similar performance
regression with llama-bench on subsequent runs when running it across
different NUMA nodes. The reason this happened was that the existing
page cache folios of model weight file (from the previous run on a
separate NUMA node) were not getting migrated (because they were not
calling MADV_POPULATE_READ since it can cause a read of a large model
weight file into the page cache all at once).

With that in mind, do we think having something like
MADV_POPULATE_READ_NOIO make sense to address such problems? Do we have
any other usecases of this too?
Or do we see any problems with this, due to which it never existed?

(Note that I haven't yet given a thought for how it should behave for
anon memory).

-ritesh