[LSF/MM/BPF BoF Session] Numa-Aware Placement for Page Cache Pages

["Ritesh Harjani (IBM)" <ritesh.list@gmail.com>]

Hi All,

Amir insisted for this :)
> IOW, bring the Hallway session into the room, so that other people
> can participate and we can use the hallway time for gossip and
> stuffing our faces.

So, since we might have few slots available in FS breakout sessions -
here is something that I was hoping to have a discussion with you all
in hallway. However, I thought maybe it will be a good idea to initiate
this thread here, to see what do you think about this.


Linux already supports memory tiers and there are ongoing discussions around
promotion of unmapped page cache pages, which lets kernel do the right thing
for userspace page cache pages on a tiered system.

v6.17 added support for per-node global reclaim via
/sys/devices/system/node/nodeX/reclaim, which lets users perform per-node
reclaim of page cache pages. We also already have interfaces that let userspace
define the lifetime of page cache pages, such as RWF_DONTCACHE and
FADV_DONTNEED. These are increasingly useful because locally-attached DRAM is
a costly resource and we don't want unwanted page cache pollution there.
Userspace, sometimes is in a better position than the kernel to know the
workload's access pattern and whether it makes sense to drop page cache pages
once the I/O is done.

So the question is:
Do we need a userspace interface for the placement policy of page cache pages on a per file basis?

Note that we do have per-task placement policies like set_mempolicy(), but those
are too coarse and don't help if userspace wants per-fd control. mmap+mbind()
doesn't reach unmapped page cache. shared_policy per-inode works for
shmem/guest_memfd but not for other filesystems (I think so, but I maybe
wrong).

So what I would like to discuss with others is:

1. Is there a need for an interface that allows userspace to do per-fd page
   placement and maybe per-fd page migration?
2. Are there applications that need such an interface or would they benefit
   from it?
3. Even if applications may not need this today, should kernel developers start
   thinking about it now, before users start abusing some not-well-defined
   existing interface. e.g. the story of echo 1 > /proc/sys/vm/drop_caches,
   which became a production workload tool despite never being intended as
   one?

Let me know if people think that this discussion qualifies for a BoF discussion at LSFMM?
Or do you think it's a bad idea altogether, if that is the case - Then
please help me understand, why so?
Before starting to jump on the implemention of any of this - I would
like to gather feedback on what do others think?

-ritesh