[PATCH 0/2] mm: don't apply task mempolicy to unmovable kernel allocations

[PATCH 0/2] mm: don't apply task mempolicy to unmovable kernel allocations

[Gregory Price]

This series stops a task's NUMA mempolicy from steering incidental,
global kernel allocations that have no real relationship to the task.

VMA-less allocations (alloc_pages(), folio_alloc(), vmalloc(), slab
refils) - fall back to the current task's mempolicy as their only
placement hint.

This sweeps in kernel memory that is not the task's:
  - unaccounted slab
  - kernel page tables
  - driver GFP_KERNEL buffers
  - one-off global structures
  - etc

On a uniform multi-socket system this is not implicitly harmful.
On a tiered-memory system a task's interleave/bind can push these
incidental allocations onto slower tiers and cause regressions.

The memory may outlive the task, or be shared by others, yet a single
task's policy dictated where it landed.  

The fix is to only follow the task policy for allocations that are
plausibly the task's:

  - movable allocations (mostly user data), and
  - allocations explicitly tied to the task via __GFP_ACCOUNT.

Everything else (unmovable and unaccounted) prefers node-local.

Cpuset still enforces any hard confinements (ALLOC_CPUSET), and
fallback allocations may still cause spillage, but this at least
prevents interleave policies from making poor placements.

Patch 1 makes page-cache placement explicit in filemap to retain
existing behavior (pagecache and metadata still end up following
the task mempolicy).  Since the metadata can be significant on
some systems, retaining this behavior will ensure there are no
surprises for existing users.

If we want to change this behavior, this patch is droppable.

Patch 2 adds the alloc_task_policy() filter for bare allocations.

Test 1: Page cache follows the task policy (unchanged)
======
A process running on node0 but bound to node1:
   (numactl --cpunodebind=0 --membind=1) writes a 1.2G file.

  membind=1 (non-local):  FilePages  node0 +0MB     node1 +1200MB
  membind=0 (control):    FilePages  node0 +1200MB  node1 +0MB 


Test 2: Incidental kernel allocations no longer follow it (changed)
======
Added a debugfs interface to do movable and kernel allocations:

 w/ --cpunodebind=0
 numactl ... --membind=1      echo 100000 > .../alloc_kernel
 numactl ... --membind=1      echo 100000 > .../alloc_movable
 numactl ... --interleave=all echo 100000 > .../alloc_kernel

                       GFP_KERNEL          GFP_HIGHUSER_MOVABLE
membind=1    base:     node1 (follows)     node1
             patched:  node0 (local)       node1 (follows)

interleave   base:     50/50 (follows)     50/50
             patched:  node0 (local)       50/50 (follows)

Movable (user) allocations are unaffected in every case.

The unmovable unaccounted kernel allocations stop following the
task policy and place node-local. With no policy set, both place
node-local as before.

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>

Gregory Price (2):
  mm/filemap: place page-cache folios via an explicit mempolicy
  mm/mempolicy: skip task mempolicy for unmovable unaccounted kernel
    allocations

 mm/filemap.c   |  5 ++++-
 mm/mempolicy.c | 40 ++++++++++++++++++++++++++++------------
 2 files changed, 32 insertions(+), 13 deletions(-)

--
2.53.0-Meta

[PATCH 1/2] mm/filemap: place page-cache folios via an explicit mempolicy

[Gregory Price]

Page-cache allocations have no mempolicy of their own and implicitly
follow first-touch task mempolicy via the page allocator.

Codify this preference explicitly in filemap.c, rather than taking an
implicit behavior from the page allocator.  No functional change; it
keeps pagecache placement stable across the following change.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Gregory Price <gourry@gourry.net>
---
 mm/filemap.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 5af62e6abca5..d419d09a58c1 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -25,6 +25,7 @@
 #include <linux/syscalls.h>
 #include <linux/mman.h>
 #include <linux/pagemap.h>
+#include <linux/mempolicy.h>
 #include <linux/file.h>
 #include <linux/uio.h>
 #include <linux/error-injection.h>
@@ -1010,7 +1011,9 @@ struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order,
 
 		return folio;
 	}
-	return folio_alloc_noprof(gfp, order);
+	/* page cache defaults to first-accessor task mempolicy for placement */
+	return folio_alloc_mpol_noprof(gfp, order, get_task_policy(current),
+			NO_INTERLEAVE_INDEX, numa_node_id());
 }
 EXPORT_SYMBOL(filemap_alloc_folio_noprof);
 #endif
-- 
2.53.0-Meta

[PATCH 2/2] mm/mempolicy: skip task mempolicy for unmovable unaccounted kernel allocations

[Gregory Price]

Bare calls to alloc_pages / folio_alloc / vmalloc / slab-refill cause
vma-less allocations which fall back to the current task's mempolicy
for a placement hint.  This has the effect of directing incidental,
global, task-unrelated allocations to follow the task's mempolicy.

On uniform multi-socket systems, this is not implicitly harmful.
On tiered memory systems, these incidental allocations can land
on slower memory tiers and actively cause performance regressions.

For example, the code in lib/stackdepot.c allocates a global
resource that sticks around for the life of the system:

  stack_depot_save_flags()
      page = alloc_pages(gfp_nested_mask(alloc_flags));

In this example:
  1) the ownership is global (stackdepot, no association to the task)
  2) The lifetime is forever (the pool is more or less permanent)
  3) Whichever task causes the FIRST allocation to occur dictates
     placement for future users - permanently.
  4) It's unaccounted and unmovable.

While this doesn't prevent fallback allocations occurring and landing
in the same class of memory the same way, a per-task interleave policy
shouldn't dictate placement for such allocations.

Make a new helper, alloc_task_policy(), to identify three classes of
allocations we explicitly do not want to follow the task policy.

 1) in_interrupt() - interrupt allocations are not related to the task
 2) __GFP_THISNODE - caller explicit placement
 3) unmovable, unaccounted kernel allocations
    !(__GFP_MOVABLE | __GFP_ACCOUNT)

The third filter keeps incidental global kernel allocations from being
spread throughout the system when a task has set a mempolicy. The other
two are existing filters consolidated by the helper.

Allowing movable allocations to follow task policy retains existing
non-harmful placement behavior, and cpuset constraints are still
enforced by ALLOC_CPUSET.  Pagecache placement is made explicit by
the preceding patch.

Everything else (unmovable and unaccounted) falls back to node-local.

User-space folios are unaffected: they fall through folio_alloc_mpol().

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Link: https://lore.kernel.org/linux-mm/akU2_Mt_JDCxmnc7@cmpxchg.org/
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Gregory Price <gourry@gourry.net>
---
 mm/mempolicy.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index fc9051e9558b..6c00dc32c8b4 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2737,16 +2737,35 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct
 }
 EXPORT_SYMBOL(vma_alloc_folio_noprof);
 
-struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order)
+/*
+ * Bare alloc_pages() calls only have task policy as a placement hint.
+ * There are 3 scenarios where we don't want to use the task policy:
+ *  1) interrupt context - these allocations are not related to the task
+ *  2) __GFP_THISNODE - the caller asked for explicit placement
+ *  3) Unmovable (!__GFP_MOVABLE), unaccounted (!__GFP_ACCOUNT) kernel memory
+ *
+ * The third filter keeps incidental global kernel allocations from being
+ * spread throughout the system when a task has set an interleave policy.
+ * This can include cached global metadata for drivers and internal
+ * services that outlives the life of the task.
+ *
+ * Allowing movable allocations to follow task policy retains existing
+ * non-harmful behavior, and cpuset constraints are still respected.
+ *
+ * No reference counting needed for current->mempolicy nor default_policy
+ */
+static struct mempolicy *alloc_task_policy(gfp_t gfp)
 {
-	struct mempolicy *pol = &default_policy;
+	if (in_interrupt() || (gfp & __GFP_THISNODE))
+		return &default_policy;
+	if (!(gfp & (__GFP_MOVABLE | __GFP_ACCOUNT)))
+		return &default_policy;
+	return get_task_policy(current);
+}
 
-	/*
-	 * No reference counting needed for current->mempolicy
-	 * nor system default_policy
-	 */
-	if (!in_interrupt() && !(gfp & __GFP_THISNODE))
-		pol = get_task_policy(current);
+struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order)
+{
+	struct mempolicy *pol = alloc_task_policy(gfp);
 
 	return alloc_pages_mpol(gfp, order, pol, NO_INTERLEAVE_INDEX,
 				       numa_node_id());
@@ -2964,13 +2983,10 @@ static unsigned long alloc_pages_bulk_preferred_many(gfp_t gfp, int nid,
 unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
 		unsigned long nr_pages, struct page **page_array)
 {
-	struct mempolicy *pol = &default_policy;
+	struct mempolicy *pol = alloc_task_policy(gfp);
 	nodemask_t *nodemask;
 	int nid;
 
-	if (!in_interrupt() && !(gfp & __GFP_THISNODE))
-		pol = get_task_policy(current);
-
 	if (pol->mode == MPOL_INTERLEAVE)
 		return alloc_pages_bulk_interleave(gfp, pol,
 							 nr_pages, page_array);
-- 
2.53.0-Meta

Re: [PATCH 0/2] mm: don't apply task mempolicy to unmovable kernel allocations

[Huang, Ying]

Gregory Price <gourry@gourry.net> writes:

> This series stops a task's NUMA mempolicy from steering incidental,
> global kernel allocations that have no real relationship to the task.
>
> VMA-less allocations (alloc_pages(), folio_alloc(), vmalloc(), slab
> refils) - fall back to the current task's mempolicy as their only
> placement hint.
>
> This sweeps in kernel memory that is not the task's:
>   - unaccounted slab
>   - kernel page tables
>   - driver GFP_KERNEL buffers
>   - one-off global structures
>   - etc
>
> On a uniform multi-socket system this is not implicitly harmful.
> On a tiered-memory system a task's interleave/bind can push these
> incidental allocations onto slower tiers and cause regressions.
>
> The memory may outlive the task, or be shared by others, yet a single
> task's policy dictated where it landed.  
>
> The fix is to only follow the task policy for allocations that are
> plausibly the task's:
>
>   - movable allocations (mostly user data), and
>   - allocations explicitly tied to the task via __GFP_ACCOUNT.
>
> Everything else (unmovable and unaccounted) prefers node-local.
>
> Cpuset still enforces any hard confinements (ALLOC_CPUSET), and
> fallback allocations may still cause spillage, but this at least
> prevents interleave policies from making poor placements.
>
> Patch 1 makes page-cache placement explicit in filemap to retain
> existing behavior (pagecache and metadata still end up following
> the task mempolicy).  Since the metadata can be significant on
> some systems, retaining this behavior will ensure there are no
> surprises for existing users.
>
> If we want to change this behavior, this patch is droppable.
>
> Patch 2 adds the alloc_task_policy() filter for bare allocations.
>
> Test 1: Page cache follows the task policy (unchanged)
> ======
> A process running on node0 but bound to node1:
>    (numactl --cpunodebind=0 --membind=1) writes a 1.2G file.
>
>   membind=1 (non-local):  FilePages  node0 +0MB     node1 +1200MB
>   membind=0 (control):    FilePages  node0 +1200MB  node1 +0MB 
>
>
> Test 2: Incidental kernel allocations no longer follow it (changed)
> ======
> Added a debugfs interface to do movable and kernel allocations:
>
>  w/ --cpunodebind=0
>  numactl ... --membind=1      echo 100000 > .../alloc_kernel
>  numactl ... --membind=1      echo 100000 > .../alloc_movable
>  numactl ... --interleave=all echo 100000 > .../alloc_kernel
>
>                        GFP_KERNEL          GFP_HIGHUSER_MOVABLE
> membind=1    base:     node1 (follows)     node1
>              patched:  node0 (local)       node1 (follows)
>
> interleave   base:     50/50 (follows)     50/50
>              patched:  node0 (local)       50/50 (follows)
>
> Movable (user) allocations are unaffected in every case.
>
> The unmovable unaccounted kernel allocations stop following the
> task policy and place node-local. With no policy set, both place
> node-local as before.

Personally, I think this should be the right thing to do theoretically.
However, you may need to find some practical issues that this resolves.

> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
>
> Gregory Price (2):
>   mm/filemap: place page-cache folios via an explicit mempolicy
>   mm/mempolicy: skip task mempolicy for unmovable unaccounted kernel
>     allocations
>
>  mm/filemap.c   |  5 ++++-
>  mm/mempolicy.c | 40 ++++++++++++++++++++++++++++------------
>  2 files changed, 32 insertions(+), 13 deletions(-)

---
Best Regards,
Huang, Ying

Re: [PATCH 0/2] mm: don't apply task mempolicy to unmovable kernel allocations

[Gregory Price]

On Thu, Jul 09, 2026 at 07:09:29PM +0800, Huang, Ying wrote:
> Gregory Price <gourry@gourry.net> writes:
> 
> > The unmovable unaccounted kernel allocations stop following the
> > task policy and place node-local. With no policy set, both place
> > node-local as before.
> 
> Personally, I think this should be the right thing to do theoretically.
> However, you may need to find some practical issues that this resolves.
> 

I don't entirely disagree, but there is at least one scenario where this
is an obvious improvement:

  BIOS-configured CXL memory brought up in ZONE_NORMAL. 

An task interleave policy on such a system will end up with unaccounted
kernel allocations landing on the remote node, which is just not
preferable at all (and uncorrectable).

It's not a complete fix (fallbacks can still occur under pressure), but
it's one piece of the puzzle.


Forward looking:  This patch makes private-node's with ZONE_NORMAL
reliably hot-un-pluggable.  But that improvement is obviously predicated
on work that isn't upstream (yet :] ).

I need to send a v2 of this with SLAB_ACCOUNT fixed up, and some numbers
to justify dropping the pagecache fix.  But I will probably sandbag this
a bit until i finally send out v5 of private nodes.

~Gregory