[PATCH] mm/memory-tiers: cache top tier nodes

[PATCH] mm/memory-tiers: cache top tier nodes

[Joshua Hahn]

node_is_toptier() is called in a few hot paths: task_numa_fault(),
should_numa_migrate_memory(), folio_migrate_flags(), etc. Each call
takes an RCU read section and performs the tier distance check again.

Tieredness for all nodes only changes on memory node hotplugs. Instead
of recomputing toptier nodes inside each of the hot paths above, compute
it only on memory node hotplug events and cache the results.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/memory-tiers.c | 36 ++++++++++--------------------------
 1 file changed, 10 insertions(+), 26 deletions(-)

diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 54851d8a195b0..2e6e02ec1fce4 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -71,6 +71,7 @@ bool folio_use_access_time(struct folio *folio)
 
 #ifdef CONFIG_NUMA_MIGRATION
 static int top_tier_adistance;
+static nodemask_t toptier_nodes __read_mostly = NODE_MASK_ALL;
 /*
  * node_demotion[] examples:
  *
@@ -276,27 +277,7 @@ static struct memory_tier *__node_get_memory_tier(int node)
 #ifdef CONFIG_NUMA_MIGRATION
 bool node_is_toptier(int node)
 {
-	bool toptier;
-	pg_data_t *pgdat;
-	struct memory_tier *memtier;
-
-	pgdat = NODE_DATA(node);
-	if (!pgdat)
-		return false;
-
-	rcu_read_lock();
-	memtier = rcu_dereference(pgdat->memtier);
-	if (!memtier) {
-		toptier = true;
-		goto out;
-	}
-	if (memtier->adistance_start <= top_tier_adistance)
-		toptier = true;
-	else
-		toptier = false;
-out:
-	rcu_read_unlock();
-	return toptier;
+	return node_isset(node, toptier_nodes);
 }
 
 void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
@@ -497,19 +478,22 @@ static void establish_demotion_targets(void)
 		}
 	}
 	/*
-	 * Now build the lower_tier mask for each node collecting node mask from
-	 * all memory tier below it. This allows us to fallback demotion page
-	 * allocation to a set of nodes that is closer the above selected
-	 * preferred node.
+	 * A node stays toptier unless it belongs to a tier below
+	 * top_tier_adistance, while each tier's lower_tier_mask collects the
+	 * nodes of every tier below it so demotion page allocation can fall
+	 * back to nodes closer to the selected preferred node.
 	 */
+	toptier_nodes = node_states[N_MEMORY];
 	lower_tier = node_states[N_MEMORY];
 	list_for_each_entry(memtier, &memory_tiers, list) {
+		tier_nodes = get_memtier_nodemask(memtier);
+		if (memtier->adistance_start > top_tier_adistance)
+			nodes_andnot(toptier_nodes, toptier_nodes, tier_nodes);
 		/*
 		 * Keep removing current tier from lower_tier nodes,
 		 * This will remove all nodes in current and above
 		 * memory tier from the lower_tier mask.
 		 */
-		tier_nodes = get_memtier_nodemask(memtier);
 		nodes_andnot(lower_tier, lower_tier, tier_nodes);
 		memtier->lower_tier_mask = lower_tier;
 	}
-- 
2.53.0-Meta

Re: [PATCH] mm/memory-tiers: cache top tier nodes

[Aneesh Kumar K.V]

Joshua Hahn <joshua.hahnjy@gmail.com> writes:

> node_is_toptier() is called in a few hot paths: task_numa_fault(),
> should_numa_migrate_memory(), folio_migrate_flags(), etc. Each call
> takes an RCU read section and performs the tier distance check again.
>
> Tieredness for all nodes only changes on memory node hotplugs. Instead
> of recomputing toptier nodes inside each of the hot paths above, compute
> it only on memory node hotplug events and cache the results.
>
> Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> ---
>  mm/memory-tiers.c | 36 ++++++++++--------------------------
>  1 file changed, 10 insertions(+), 26 deletions(-)
>
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 54851d8a195b0..2e6e02ec1fce4 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -71,6 +71,7 @@ bool folio_use_access_time(struct folio *folio)
>  
>  #ifdef CONFIG_NUMA_MIGRATION
>  static int top_tier_adistance;
> +static nodemask_t toptier_nodes __read_mostly = NODE_MASK_ALL;
>  /*
>   * node_demotion[] examples:
>   *
> @@ -276,27 +277,7 @@ static struct memory_tier *__node_get_memory_tier(int node)
>  #ifdef CONFIG_NUMA_MIGRATION
>  bool node_is_toptier(int node)
>  {
> -	bool toptier;
> -	pg_data_t *pgdat;
> -	struct memory_tier *memtier;
> -
> -	pgdat = NODE_DATA(node);
> -	if (!pgdat)
> -		return false;
> -
> -	rcu_read_lock();
> -	memtier = rcu_dereference(pgdat->memtier);
> -	if (!memtier) {
> -		toptier = true;
> -		goto out;
> -	}
> -	if (memtier->adistance_start <= top_tier_adistance)
> -		toptier = true;
> -	else
> -		toptier = false;
> -out:
> -	rcu_read_unlock();
> -	return toptier;
> +	return node_isset(node, toptier_nodes);
>  }
>  
>  void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
> @@ -497,19 +478,22 @@ static void establish_demotion_targets(void)
>  		}
>  	}
>  	/*
> -	 * Now build the lower_tier mask for each node collecting node mask from
> -	 * all memory tier below it. This allows us to fallback demotion page
> -	 * allocation to a set of nodes that is closer the above selected
> -	 * preferred node.
> +	 * A node stays toptier unless it belongs to a tier below
> +	 * top_tier_adistance, while each tier's lower_tier_mask collects the
> +	 * nodes of every tier below it so demotion page allocation can fall
> +	 * back to nodes closer to the selected preferred node.
>  	 */
> +	toptier_nodes = node_states[N_MEMORY];
>

Won't this result in a larger window where every node_is_toptier() call
can return an incorrect result? This change would make every memory node
a top-tier node until the loop below clears them from the toptier_nodes
nodemask.

I assume an incorrect result from node_is_toptier() is not too bad, but
is the performance impact large enough to make this change?



>  	lower_tier = node_states[N_MEMORY];
>  	list_for_each_entry(memtier, &memory_tiers, list) {
> +		tier_nodes = get_memtier_nodemask(memtier);
> +		if (memtier->adistance_start > top_tier_adistance)
> +			nodes_andnot(toptier_nodes, toptier_nodes, tier_nodes);
>  		/*
>  		 * Keep removing current tier from lower_tier nodes,
>  		 * This will remove all nodes in current and above
>  		 * memory tier from the lower_tier mask.
>  		 */
> -		tier_nodes = get_memtier_nodemask(memtier);
>  		nodes_andnot(lower_tier, lower_tier, tier_nodes);
>  		memtier->lower_tier_mask = lower_tier;
>  	}
> -- 
> 2.53.0-Meta

-aneesh

Re: [PATCH] mm/memory-tiers: cache top tier nodes

[Ritesh Harjani]

Joshua Hahn <joshua.hahnjy@gmail.com> writes:

> node_is_toptier() is called in a few hot paths: task_numa_fault(),
> should_numa_migrate_memory(), folio_migrate_flags(), etc. Each call
> takes an RCU read section and performs the tier distance check again.
>
> Tieredness for all nodes only changes on memory node hotplugs. Instead
> of recomputing toptier nodes inside each of the hot paths above, compute
> it only on memory node hotplug events and cache the results.
>

Did we observe this function "node_is_toptier()" in any of the perf
reports? How much perf benefit we see with this, if at all?
It will be worth adding that info to the commit msg.


> Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> ---
>  mm/memory-tiers.c | 36 ++++++++++--------------------------
>  1 file changed, 10 insertions(+), 26 deletions(-)
>
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 54851d8a195b0..2e6e02ec1fce4 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -71,6 +71,7 @@ bool folio_use_access_time(struct folio *folio)
>  
>  #ifdef CONFIG_NUMA_MIGRATION
>  static int top_tier_adistance;
> +static nodemask_t toptier_nodes __read_mostly = NODE_MASK_ALL;
>  /*
>   * node_demotion[] examples:
>   *
> @@ -276,27 +277,7 @@ static struct memory_tier *__node_get_memory_tier(int node)
>  #ifdef CONFIG_NUMA_MIGRATION
>  bool node_is_toptier(int node)
>  {
> -	bool toptier;
> -	pg_data_t *pgdat;
> -	struct memory_tier *memtier;
> -
> -	pgdat = NODE_DATA(node);
> -	if (!pgdat)
> -		return false;
> -
> -	rcu_read_lock();
> -	memtier = rcu_dereference(pgdat->memtier);
> -	if (!memtier) {
> -		toptier = true;
> -		goto out;
> -	}
> -	if (memtier->adistance_start <= top_tier_adistance)
> -		toptier = true;
> -	else
> -		toptier = false;
> -out:
> -	rcu_read_unlock();
> -	return toptier;
> +	return node_isset(node, toptier_nodes);
>  }
>  
>  void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
> @@ -497,19 +478,22 @@ static void establish_demotion_targets(void)
>  		}
>  	}
>  	/*
> -	 * Now build the lower_tier mask for each node collecting node mask from
> -	 * all memory tier below it. This allows us to fallback demotion page
> -	 * allocation to a set of nodes that is closer the above selected
> -	 * preferred node.
> +	 * A node stays toptier unless it belongs to a tier below
> +	 * top_tier_adistance, while each tier's lower_tier_mask collects the
> +	 * nodes of every tier below it so demotion page allocation can fall
> +	 * back to nodes closer to the selected preferred node.
>  	 */
> +	toptier_nodes = node_states[N_MEMORY];
>  	lower_tier = node_states[N_MEMORY];
>  	list_for_each_entry(memtier, &memory_tiers, list) {
> +		tier_nodes = get_memtier_nodemask(memtier);
> +		if (memtier->adistance_start > top_tier_adistance)
> +			nodes_andnot(toptier_nodes, toptier_nodes, tier_nodes);

I looked into this and I don't see any major issue with this patch.

However few things worth checking are:
1. From what I understood say if we add a pmem node as system ram, then
due to the above logic, it will go through these transition states..
    0 -> 1 -> 0
This is because we update toptier_nodes to nodes_state[N_MEMORY], which
means, pmem's nid becomes top tier and then we quickly transition that
back to lower tier by clearing the mask.

Instead maybe updating toptier_nodes in one shot at the end would avoid
this intermediate state.

2. Also IIRC, after the nid is online, numa balancing for tiered memory
can always work. And we store different values (I guess in
folio->flags), depending upon whether the nid is toptier v/s lower tier.
(_last_cpupid v/s folio last access time). It's worth checking that path
once.
...So, I looked at it and I think the same transient discripancy (of
whether the node is toptier or lower tier) is true even with the
original code. So I don't think that it should be an issue.


-ritesh

>  		/*
>  		 * Keep removing current tier from lower_tier nodes,
>  		 * This will remove all nodes in current and above
>  		 * memory tier from the lower_tier mask.
>  		 */
> -		tier_nodes = get_memtier_nodemask(memtier);
>  		nodes_andnot(lower_tier, lower_tier, tier_nodes);
>  		memtier->lower_tier_mask = lower_tier;
>  	}
> -- 
> 2.53.0-Meta

Re: [PATCH] mm/memory-tiers: cache top tier nodes

[Joshua Hahn]

On Sun, 12 Jul 2026 14:13:24 +0530 Aneesh Kumar K.V <aneesh.kumar@kernel.org> wrote:

> Joshua Hahn <joshua.hahnjy@gmail.com> writes:
> 
> > node_is_toptier() is called in a few hot paths: task_numa_fault(),
> > should_numa_migrate_memory(), folio_migrate_flags(), etc. Each call
> > takes an RCU read section and performs the tier distance check again.
> >
> > Tieredness for all nodes only changes on memory node hotplugs. Instead
> > of recomputing toptier nodes inside each of the hot paths above, compute
> > it only on memory node hotplug events and cache the results.
> >
> > Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> > ---
> >  mm/memory-tiers.c | 36 ++++++++++--------------------------
> >  1 file changed, 10 insertions(+), 26 deletions(-)
> >
> > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> > index 54851d8a195b0..2e6e02ec1fce4 100644
> > --- a/mm/memory-tiers.c
> > +++ b/mm/memory-tiers.c
> > @@ -71,6 +71,7 @@ bool folio_use_access_time(struct folio *folio)
> >  
> >  #ifdef CONFIG_NUMA_MIGRATION
> >  static int top_tier_adistance;
> > +static nodemask_t toptier_nodes __read_mostly = NODE_MASK_ALL;
> >  /*
> >   * node_demotion[] examples:
> >   *
> > @@ -276,27 +277,7 @@ static struct memory_tier *__node_get_memory_tier(int node)
> >  #ifdef CONFIG_NUMA_MIGRATION
> >  bool node_is_toptier(int node)
> >  {
> > -	bool toptier;
> > -	pg_data_t *pgdat;
> > -	struct memory_tier *memtier;
> > -
> > -	pgdat = NODE_DATA(node);
> > -	if (!pgdat)
> > -		return false;
> > -
> > -	rcu_read_lock();
> > -	memtier = rcu_dereference(pgdat->memtier);
> > -	if (!memtier) {
> > -		toptier = true;
> > -		goto out;
> > -	}
> > -	if (memtier->adistance_start <= top_tier_adistance)
> > -		toptier = true;
> > -	else
> > -		toptier = false;
> > -out:
> > -	rcu_read_unlock();
> > -	return toptier;
> > +	return node_isset(node, toptier_nodes);
> >  }
> >  
> >  void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
> > @@ -497,19 +478,22 @@ static void establish_demotion_targets(void)
> >  		}
> >  	}
> >  	/*
> > -	 * Now build the lower_tier mask for each node collecting node mask from
> > -	 * all memory tier below it. This allows us to fallback demotion page
> > -	 * allocation to a set of nodes that is closer the above selected
> > -	 * preferred node.
> > +	 * A node stays toptier unless it belongs to a tier below
> > +	 * top_tier_adistance, while each tier's lower_tier_mask collects the
> > +	 * nodes of every tier below it so demotion page allocation can fall
> > +	 * back to nodes closer to the selected preferred node.
> >  	 */
> > +	toptier_nodes = node_states[N_MEMORY];

Hello Aneesh, thank you very much for your review!

> Won't this result in a larger window where every node_is_toptier() call
> can return an incorrect result? This change would make every memory node
> a top-tier node until the loop below clears them from the toptier_nodes
> nodemask.

Yes, definitely. I think I should have instead constructed a local nodemask
and then swapped it out at the end, I overlooked this part.

> I assume an incorrect result from node_is_toptier() is not too bad, but
> is the performance impact large enough to make this change?

Yes, but I think it's still important to keep it correct.
On the notion of performance impact, it's not too big in the upstream
kernel, but I'm working on a series that adds more hotpaths that
frequently checks whether a folio belongs to a toptier node or not. 
You can read more about it here [1].

Thank you again for your review, I'll send up a fixed version!
Joshua

[1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/

Re: [PATCH] mm/memory-tiers: cache top tier nodes

[Joshua Hahn]

On Tue, 14 Jul 2026 10:17:00 +0530 Ritesh Harjani (IBM) <ritesh.list@gmail.com> wrote:

> Joshua Hahn <joshua.hahnjy@gmail.com> writes:
> 
> > node_is_toptier() is called in a few hot paths: task_numa_fault(),
> > should_numa_migrate_memory(), folio_migrate_flags(), etc. Each call
> > takes an RCU read section and performs the tier distance check again.
> >
> > Tieredness for all nodes only changes on memory node hotplugs. Instead
> > of recomputing toptier nodes inside each of the hot paths above, compute
> > it only on memory node hotplug events and cache the results.
> >
> 
> Did we observe this function "node_is_toptier()" in any of the perf
> reports? How much perf benefit we see with this, if at all?
> It will be worth adding that info to the commit msg.
> 
> 
> > Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> > ---
> >  mm/memory-tiers.c | 36 ++++++++++--------------------------
> >  1 file changed, 10 insertions(+), 26 deletions(-)
> >
> > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> > index 54851d8a195b0..2e6e02ec1fce4 100644
> > --- a/mm/memory-tiers.c
> > +++ b/mm/memory-tiers.c
> > @@ -71,6 +71,7 @@ bool folio_use_access_time(struct folio *folio)
> >  
> >  #ifdef CONFIG_NUMA_MIGRATION
> >  static int top_tier_adistance;
> > +static nodemask_t toptier_nodes __read_mostly = NODE_MASK_ALL;
> >  /*
> >   * node_demotion[] examples:
> >   *
> > @@ -276,27 +277,7 @@ static struct memory_tier *__node_get_memory_tier(int node)
> >  #ifdef CONFIG_NUMA_MIGRATION
> >  bool node_is_toptier(int node)
> >  {
> > -	bool toptier;
> > -	pg_data_t *pgdat;
> > -	struct memory_tier *memtier;
> > -
> > -	pgdat = NODE_DATA(node);
> > -	if (!pgdat)
> > -		return false;
> > -
> > -	rcu_read_lock();
> > -	memtier = rcu_dereference(pgdat->memtier);
> > -	if (!memtier) {
> > -		toptier = true;
> > -		goto out;
> > -	}
> > -	if (memtier->adistance_start <= top_tier_adistance)
> > -		toptier = true;
> > -	else
> > -		toptier = false;
> > -out:
> > -	rcu_read_unlock();
> > -	return toptier;
> > +	return node_isset(node, toptier_nodes);
> >  }
> >  
> >  void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
> > @@ -497,19 +478,22 @@ static void establish_demotion_targets(void)
> >  		}
> >  	}
> >  	/*
> > -	 * Now build the lower_tier mask for each node collecting node mask from
> > -	 * all memory tier below it. This allows us to fallback demotion page
> > -	 * allocation to a set of nodes that is closer the above selected
> > -	 * preferred node.
> > +	 * A node stays toptier unless it belongs to a tier below
> > +	 * top_tier_adistance, while each tier's lower_tier_mask collects the
> > +	 * nodes of every tier below it so demotion page allocation can fall
> > +	 * back to nodes closer to the selected preferred node.
> >  	 */
> > +	toptier_nodes = node_states[N_MEMORY];
> >  	lower_tier = node_states[N_MEMORY];
> >  	list_for_each_entry(memtier, &memory_tiers, list) {
> > +		tier_nodes = get_memtier_nodemask(memtier);
> > +		if (memtier->adistance_start > top_tier_adistance)
> > +			nodes_andnot(toptier_nodes, toptier_nodes, tier_nodes);

Hello Ritesh,

Thank you very much for your review!

> I looked into this and I don't see any major issue with this patch.
> 
> However few things worth checking are:
> 1. From what I understood say if we add a pmem node as system ram, then
> due to the above logic, it will go through these transition states..
>     0 -> 1 -> 0
> This is because we update toptier_nodes to nodes_state[N_MEMORY], which
> means, pmem's nid becomes top tier and then we quickly transition that
> back to lower tier by clearing the mask.
> Instead maybe updating toptier_nodes in one shot at the end would avoid
> this intermediate state.

Yes, this is true. One thing that I think could be a problem for that is
there is not a really good way to make the transition, since nodemask_t
can be wider than a single word and I think that would make the
transitions have to happen in word chunks, which could leave the nodemask
in a partially torn state. data_race() works but it does not capture
the nuance that node_is_toptier will always only check one bit at a time
so I'm actually not entirely sure what the best approach is.

> 2. Also IIRC, after the nid is online, numa balancing for tiered memory
> can always work. And we store different values (I guess in
> folio->flags), depending upon whether the nid is toptier v/s lower tier.
> (_last_cpupid v/s folio last access time). It's worth checking that path
> once.
> ...So, I looked at it and I think the same transient discripancy (of
> whether the node is toptier or lower tier) is true even with the
> original code. So I don't think that it should be an issue.

Thank you for checking! I think I'll need to think about the above concern
for a bit more. I'll also profile whether this is really even a performance
concern or not. I just thought it would be good to cache this
calculation since it's such a rare operation to update which node is
toptier or not, and it tends not to happen when there is much happening
on the system anyways.

Thank you again for the review, have a great day!
Joshua