[PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Vlastimil Babka (SUSE)]

A customer has reported excessive reclaim activity despite memory above
high watermarks. Analyzing support data (including sar log) revealed the
following scenario:

- the system has almost all memory reserved as 1G hugepages for kvm
  guests, and operates normally
- due to a spike of memory pressure from userspace, highatomic
  reservations occur (probably due to e.g. concurrent network receive
  activity)
- on one of the 8 NUMA nodes, this creates a situation where free pages
  are above high watermark, but when __zone_watermark_unusable_free()
  considers highatomic pageblocks, the result appears to be below the
  low watermark
- pgdat_balanced() is false, kswapd hopelessly keeps trying to balance,
  spending cpu time and disrupting the system. It has nothing to reclaim
  because the node is filled with hugetlbfs hugepages.
- since the overal memory pressure is under control again and there are
  no allocations restricted to the affected node, they are satisfied
  from other nodes without going through direct reclaim
- since direct reclaim is the only code that calls
  unreserve_highatomic_pageblock(), this never happens and nothing fixes
  the situation

Note this was observed on a 6.4-based kernel and there have been recent
fixes upstream limiting the highatomic reservation side, but I don't
think they can prevent the same happening in mainline, only perhaps
lower the chances. Nothing seems to prevent the issue completely.

For example, while we limit the amount of highatomic pageblocks to 1% of
the zone, this calculation does not subtract the amount of hugepages or
otherwise permanent allocations, so the highatomic pageblocks can end up
taking much higher share of actually reclaimable memory.

The most targeted fix thus appears to be to allow kswapd perform the
highatomic pageblock unreserve to match direct reclaim. It's also
somewhat similar in principle to how kswapd reduces watermark boosting.

Hook this highatomic unreserve to the moment we are increasing kswapd
reclaim priority, as that is when kswapd is struggling to make progress
and it should roughly match the case of direct reclaim failure.

The only caveat is that direct reclaim knows which migratetype it wants
to allocate and thus changes the highatomic pageblock to. Kswapd doesn't
track that, so use MIGRATE_UNMOVABLE. Highatomic allocations usually are
not movable, and this will avoid creating nominally movable pageblocks
polluted with unmovable pages, so it should be the least problematic
default.

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
We interrupt the regular stream of AI fixes by something a bit
different. RFC to get agreement on the approach. Haven't tried
reproducing the issue in mainline, but it seems quite clear to me from
the data I've seen.
---
 mm/page_alloc.c | 126 +++++++++++++++++++++++++++++---------------------------
 mm/page_alloc.h |   3 ++
 mm/vmscan.c     |  16 ++++++-
 3 files changed, 83 insertions(+), 62 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a8cac5e01066..785a13570788 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3476,6 +3476,68 @@ static void reserve_highatomic_pageblock(struct page *page, int order,
 	}
 }
 
+bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
+					 bool force)
+{
+	/*
+	 * Preserve at least one pageblock unless memory pressure
+	 * is really high.
+	 */
+	if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages)
+		return false;
+
+	guard(spinlock_irqsave)(&zone->lock);
+	for (int order = 0; order < NR_PAGE_ORDERS; order++) {
+		struct free_area *area = &(zone->free_area[order]);
+		unsigned long size;
+		struct page *page;
+		int ret;
+
+		page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
+		if (!page)
+			continue;
+
+		size = max(pageblock_nr_pages, 1UL << order);
+		/*
+		 * It should never happen but changes to locking could
+		 * inadvertently allow a per-cpu drain to add pages to
+		 * MIGRATE_HIGHATOMIC while unreserving so be safe and watch for
+		 * underflows.
+		 */
+		if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
+			size = zone->nr_reserved_highatomic;
+		zone->nr_reserved_highatomic -= size;
+
+		/*
+		 * Convert to migratetype and avoid the normal pageblock
+		 * stealing heuristics. Minimally, the caller is doing the work
+		 * and needs the pages. More importantly, if the block was
+		 * always converted to MIGRATE_UNMOVABLE or another type then
+		 * the number of pageblocks that cannot be completely freed may
+		 * increase.
+		 */
+		if (order < pageblock_order)
+			ret = move_freepages_block(zone, page,
+					MIGRATE_HIGHATOMIC, migratetype);
+		else {
+			move_to_free_list(page, zone, order,
+					  MIGRATE_HIGHATOMIC,
+					  migratetype);
+			change_pageblock_range(page, order, migratetype);
+			ret = 1;
+		}
+		/*
+		 * Reserving the block(s) already succeeded,
+		 * so this should not fail on zone boundaries.
+		 */
+		WARN_ON_ONCE(ret == -1);
+		if (ret > 0)
+			return true;
+	}
+
+	return false;
+}
+
 /*
  * Used when an allocation is about to fail under memory pressure. This
  * potentially hurts the reliability of high-order allocations when under
@@ -3491,70 +3553,12 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,
 	struct zonelist *zonelist = ac->zonelist;
 	struct zoneref *z;
 	struct zone *zone;
-	struct page *page;
-	int order;
-	int ret;
 
 	for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx,
 								ac->nodemask) {
-		/*
-		 * Preserve at least one pageblock unless memory pressure
-		 * is really high.
-		 */
-		if (!force && zone->nr_reserved_highatomic <=
-					pageblock_nr_pages)
-			continue;
-
-		guard(spinlock_irqsave)(&zone->lock);
-		for (order = 0; order < NR_PAGE_ORDERS; order++) {
-			struct free_area *area = &(zone->free_area[order]);
-			unsigned long size;
-
-			page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
-			if (!page)
-				continue;
-
-			size = max(pageblock_nr_pages, 1UL << order);
-			/*
-			 * It should never happen but changes to
-			 * locking could inadvertently allow a per-cpu
-			 * drain to add pages to MIGRATE_HIGHATOMIC
-			 * while unreserving so be safe and watch for
-			 * underflows.
-			 */
-			if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
-				size = zone->nr_reserved_highatomic;
-			zone->nr_reserved_highatomic -= size;
-
-			/*
-			 * Convert to ac->migratetype and avoid the normal
-			 * pageblock stealing heuristics. Minimally, the caller
-			 * is doing the work and needs the pages. More
-			 * importantly, if the block was always converted to
-			 * MIGRATE_UNMOVABLE or another type then the number
-			 * of pageblocks that cannot be completely freed
-			 * may increase.
-			 */
-			if (order < pageblock_order)
-				ret = move_freepages_block(zone, page,
-							   MIGRATE_HIGHATOMIC,
-							   ac->migratetype);
-			else {
-				move_to_free_list(page, zone, order,
-						  MIGRATE_HIGHATOMIC,
-						  ac->migratetype);
-				change_pageblock_range(page, order,
-						       ac->migratetype);
-				ret = 1;
-			}
-			/*
-			 * Reserving the block(s) already succeeded,
-			 * so this should not fail on zone boundaries.
-			 */
-			WARN_ON_ONCE(ret == -1);
-			if (ret > 0)
-				return ret;
-		}
+		if (unreserve_highatomic_pageblock_zone(zone,
+							ac->migratetype, force))
+			return true;
 	}
 
 	return false;
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index e238ee91ed24..9fba6480b18d 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -282,6 +282,9 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)
 	return list_empty(&area->free_list[migratetype]);
 }
 
+bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
+					 bool force);
+
 /* Convert GFP flags to their corresponding migrate type */
 #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
 #define GFP_MOVABLE_SHIFT 3
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..91c622068d99 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
 		if (nr_boost_reclaim && !nr_reclaimed)
 			break;
 
-		if (raise_priority || !nr_reclaimed)
+		if (raise_priority || !nr_reclaimed) {
+			/*
+			 * Similarly to when direct reclaim is retrying, try to
+			 * unreserve highatomic pageblocks on the node. Convert
+			 * them to movable ones as kswapd doesn't have a target
+			 * migratetype, and passing unmovable should lead to
+			 * least permanent fragmentation.
+			 */
+			for (int i = 0; i <= highest_zoneidx; i++) {
+				struct zone *zone = pgdat->node_zones + i;
+
+				unreserve_highatomic_pageblock_zone(zone,
+						MIGRATE_UNMOVABLE, false);
+			}
 			sc.priority--;
+		}
 	} while (sc.priority >= 1);
 
 	/*

---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261030-kswapd-highatomic-unreserve-7e6629ec65d0

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Thadeu Lima de Souza Cascardo]

On Wed, Nov 04, 2026 at 07:38:44PM +0100, Vlastimil Babka (SUSE) wrote:
> A customer has reported excessive reclaim activity despite memory above
> high watermarks. Analyzing support data (including sar log) revealed the
> following scenario:
> 
> - the system has almost all memory reserved as 1G hugepages for kvm
>   guests, and operates normally
> - due to a spike of memory pressure from userspace, highatomic
>   reservations occur (probably due to e.g. concurrent network receive
>   activity)

I think you might mean 6a204d4b14c9 ("mm/page_alloc: only set
ALLOC_HIGHATOMIC for __GPF_HIGH allocations") as one of the fixes you
mention below.

In that case, GFP_NOWAIT (which is typical for network activity, AFAIR)
would be one candidate that would produce such reservations. But I agree
that if any other real highatomic allocation burst would happen, it might
produce such reservations.

> - on one of the 8 NUMA nodes, this creates a situation where free pages
>   are above high watermark, but when __zone_watermark_unusable_free()
>   considers highatomic pageblocks, the result appears to be below the
>   low watermark
> - pgdat_balanced() is false, kswapd hopelessly keeps trying to balance,
>   spending cpu time and disrupting the system. It has nothing to reclaim
>   because the node is filled with hugetlbfs hugepages.
> - since the overal memory pressure is under control again and there are
>   no allocations restricted to the affected node, they are satisfied
>   from other nodes without going through direct reclaim
> - since direct reclaim is the only code that calls
>   unreserve_highatomic_pageblock(), this never happens and nothing fixes
>   the situation
> 

I am trying to understand what is going on here: so allocations on that
node are only !__GFP_DIRECT_RECLAIM? That would explain the highatomic
reservations in the presence of the bug just mentioned.

> Note this was observed on a 6.4-based kernel and there have been recent
> fixes upstream limiting the highatomic reservation side, but I don't
> think they can prevent the same happening in mainline, only perhaps
> lower the chances. Nothing seems to prevent the issue completely.
> 
> For example, while we limit the amount of highatomic pageblocks to 1% of
> the zone, this calculation does not subtract the amount of hugepages or
> otherwise permanent allocations, so the highatomic pageblocks can end up
> taking much higher share of actually reclaimable memory.
> 
> The most targeted fix thus appears to be to allow kswapd perform the
> highatomic pageblock unreserve to match direct reclaim. It's also
> somewhat similar in principle to how kswapd reduces watermark boosting.
> 
> Hook this highatomic unreserve to the moment we are increasing kswapd
> reclaim priority, as that is when kswapd is struggling to make progress
> and it should roughly match the case of direct reclaim failure.
> 

I tried considering a different approach, making the allocation do the
unreserve in the absence of __GFP_DIRECT_RECLAIM, but I don't see why not
let kswapd do it.

Regards.
Cascardo.

> The only caveat is that direct reclaim knows which migratetype it wants
> to allocate and thus changes the highatomic pageblock to. Kswapd doesn't
> track that, so use MIGRATE_UNMOVABLE. Highatomic allocations usually are
> not movable, and this will avoid creating nominally movable pageblocks
> polluted with unmovable pages, so it should be the least problematic
> default.
> 
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
> We interrupt the regular stream of AI fixes by something a bit
> different. RFC to get agreement on the approach. Haven't tried
> reproducing the issue in mainline, but it seems quite clear to me from
> the data I've seen.
> ---
>  mm/page_alloc.c | 126 +++++++++++++++++++++++++++++---------------------------
>  mm/page_alloc.h |   3 ++
>  mm/vmscan.c     |  16 ++++++-
>  3 files changed, 83 insertions(+), 62 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a8cac5e01066..785a13570788 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3476,6 +3476,68 @@ static void reserve_highatomic_pageblock(struct page *page, int order,
>  	}
>  }
>  
> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
> +					 bool force)
> +{
> +	/*
> +	 * Preserve at least one pageblock unless memory pressure
> +	 * is really high.
> +	 */
> +	if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages)
> +		return false;
> +
> +	guard(spinlock_irqsave)(&zone->lock);
> +	for (int order = 0; order < NR_PAGE_ORDERS; order++) {
> +		struct free_area *area = &(zone->free_area[order]);
> +		unsigned long size;
> +		struct page *page;
> +		int ret;
> +
> +		page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
> +		if (!page)
> +			continue;
> +
> +		size = max(pageblock_nr_pages, 1UL << order);
> +		/*
> +		 * It should never happen but changes to locking could
> +		 * inadvertently allow a per-cpu drain to add pages to
> +		 * MIGRATE_HIGHATOMIC while unreserving so be safe and watch for
> +		 * underflows.
> +		 */
> +		if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
> +			size = zone->nr_reserved_highatomic;
> +		zone->nr_reserved_highatomic -= size;
> +
> +		/*
> +		 * Convert to migratetype and avoid the normal pageblock
> +		 * stealing heuristics. Minimally, the caller is doing the work
> +		 * and needs the pages. More importantly, if the block was
> +		 * always converted to MIGRATE_UNMOVABLE or another type then
> +		 * the number of pageblocks that cannot be completely freed may
> +		 * increase.
> +		 */
> +		if (order < pageblock_order)
> +			ret = move_freepages_block(zone, page,
> +					MIGRATE_HIGHATOMIC, migratetype);
> +		else {
> +			move_to_free_list(page, zone, order,
> +					  MIGRATE_HIGHATOMIC,
> +					  migratetype);
> +			change_pageblock_range(page, order, migratetype);
> +			ret = 1;
> +		}
> +		/*
> +		 * Reserving the block(s) already succeeded,
> +		 * so this should not fail on zone boundaries.
> +		 */
> +		WARN_ON_ONCE(ret == -1);
> +		if (ret > 0)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
>  /*
>   * Used when an allocation is about to fail under memory pressure. This
>   * potentially hurts the reliability of high-order allocations when under
> @@ -3491,70 +3553,12 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,
>  	struct zonelist *zonelist = ac->zonelist;
>  	struct zoneref *z;
>  	struct zone *zone;
> -	struct page *page;
> -	int order;
> -	int ret;
>  
>  	for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx,
>  								ac->nodemask) {
> -		/*
> -		 * Preserve at least one pageblock unless memory pressure
> -		 * is really high.
> -		 */
> -		if (!force && zone->nr_reserved_highatomic <=
> -					pageblock_nr_pages)
> -			continue;
> -
> -		guard(spinlock_irqsave)(&zone->lock);
> -		for (order = 0; order < NR_PAGE_ORDERS; order++) {
> -			struct free_area *area = &(zone->free_area[order]);
> -			unsigned long size;
> -
> -			page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
> -			if (!page)
> -				continue;
> -
> -			size = max(pageblock_nr_pages, 1UL << order);
> -			/*
> -			 * It should never happen but changes to
> -			 * locking could inadvertently allow a per-cpu
> -			 * drain to add pages to MIGRATE_HIGHATOMIC
> -			 * while unreserving so be safe and watch for
> -			 * underflows.
> -			 */
> -			if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
> -				size = zone->nr_reserved_highatomic;
> -			zone->nr_reserved_highatomic -= size;
> -
> -			/*
> -			 * Convert to ac->migratetype and avoid the normal
> -			 * pageblock stealing heuristics. Minimally, the caller
> -			 * is doing the work and needs the pages. More
> -			 * importantly, if the block was always converted to
> -			 * MIGRATE_UNMOVABLE or another type then the number
> -			 * of pageblocks that cannot be completely freed
> -			 * may increase.
> -			 */
> -			if (order < pageblock_order)
> -				ret = move_freepages_block(zone, page,
> -							   MIGRATE_HIGHATOMIC,
> -							   ac->migratetype);
> -			else {
> -				move_to_free_list(page, zone, order,
> -						  MIGRATE_HIGHATOMIC,
> -						  ac->migratetype);
> -				change_pageblock_range(page, order,
> -						       ac->migratetype);
> -				ret = 1;
> -			}
> -			/*
> -			 * Reserving the block(s) already succeeded,
> -			 * so this should not fail on zone boundaries.
> -			 */
> -			WARN_ON_ONCE(ret == -1);
> -			if (ret > 0)
> -				return ret;
> -		}
> +		if (unreserve_highatomic_pageblock_zone(zone,
> +							ac->migratetype, force))
> +			return true;
>  	}
>  
>  	return false;
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index e238ee91ed24..9fba6480b18d 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -282,6 +282,9 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)
>  	return list_empty(&area->free_list[migratetype]);
>  }
>  
> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
> +					 bool force);
> +
>  /* Convert GFP flags to their corresponding migrate type */
>  #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
>  #define GFP_MOVABLE_SHIFT 3
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 9959ca158ce8..91c622068d99 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
>  		if (nr_boost_reclaim && !nr_reclaimed)
>  			break;
>  
> -		if (raise_priority || !nr_reclaimed)
> +		if (raise_priority || !nr_reclaimed) {
> +			/*
> +			 * Similarly to when direct reclaim is retrying, try to
> +			 * unreserve highatomic pageblocks on the node. Convert
> +			 * them to movable ones as kswapd doesn't have a target
> +			 * migratetype, and passing unmovable should lead to
> +			 * least permanent fragmentation.
> +			 */
> +			for (int i = 0; i <= highest_zoneidx; i++) {
> +				struct zone *zone = pgdat->node_zones + i;
> +
> +				unreserve_highatomic_pageblock_zone(zone,
> +						MIGRATE_UNMOVABLE, false);
> +			}
>  			sc.priority--;
> +		}
>  	} while (sc.priority >= 1);
>  
>  	/*
> 
> ---
> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
> change-id: 20261030-kswapd-highatomic-unreserve-7e6629ec65d0
> 

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Vlastimil Babka (SUSE)]

On 11/4/26 21:03, Thadeu Lima de Souza Cascardo wrote:
> On Wed, Nov 04, 2026 at 07:38:44PM +0100, Vlastimil Babka (SUSE) wrote:
>> A customer has reported excessive reclaim activity despite memory above
>> high watermarks. Analyzing support data (including sar log) revealed the
>> following scenario:
>> 
>> - the system has almost all memory reserved as 1G hugepages for kvm
>>   guests, and operates normally
>> - due to a spike of memory pressure from userspace, highatomic
>>   reservations occur (probably due to e.g. concurrent network receive
>>   activity)
> 
> I think you might mean 6a204d4b14c9 ("mm/page_alloc: only set
> ALLOC_HIGHATOMIC for __GPF_HIGH allocations") as one of the fixes you
> mention below.

Indeed.

> In that case, GFP_NOWAIT (which is typical for network activity, AFAIR)
> would be one candidate that would produce such reservations. But I agree
> that if any other real highatomic allocation burst would happen, it might
> produce such reservations.

Ack.

>> - on one of the 8 NUMA nodes, this creates a situation where free pages
>>   are above high watermark, but when __zone_watermark_unusable_free()
>>   considers highatomic pageblocks, the result appears to be below the
>>   low watermark
>> - pgdat_balanced() is false, kswapd hopelessly keeps trying to balance,
>>   spending cpu time and disrupting the system. It has nothing to reclaim
>>   because the node is filled with hugetlbfs hugepages.
>> - since the overal memory pressure is under control again and there are
>>   no allocations restricted to the affected node, they are satisfied
>>   from other nodes without going through direct reclaim
>> - since direct reclaim is the only code that calls
>>   unreserve_highatomic_pageblock(), this never happens and nothing fixes
>>   the situation
>> 
> 
> I am trying to understand what is going on here: so allocations on that
> node are only !__GFP_DIRECT_RECLAIM? That would explain the highatomic
> reservations in the presence of the bug just mentioned.

They are probably __GFP_DIRECT_RECLAIM, but if they only origin on a cpu
local to the affected node but can fallback to any other (which is the usual
case) and zone_reclaim_mode is disabled (by default it is), they will just
succeed on another node and not attempt the direct reclaim.

>> Note this was observed on a 6.4-based kernel and there have been recent
>> fixes upstream limiting the highatomic reservation side, but I don't
>> think they can prevent the same happening in mainline, only perhaps
>> lower the chances. Nothing seems to prevent the issue completely.
>> 
>> For example, while we limit the amount of highatomic pageblocks to 1% of
>> the zone, this calculation does not subtract the amount of hugepages or
>> otherwise permanent allocations, so the highatomic pageblocks can end up
>> taking much higher share of actually reclaimable memory.
>> 
>> The most targeted fix thus appears to be to allow kswapd perform the
>> highatomic pageblock unreserve to match direct reclaim. It's also
>> somewhat similar in principle to how kswapd reduces watermark boosting.
>> 
>> Hook this highatomic unreserve to the moment we are increasing kswapd
>> reclaim priority, as that is when kswapd is struggling to make progress
>> and it should roughly match the case of direct reclaim failure.
>> 
> 
> I tried considering a different approach, making the allocation do the
> unreserve in the absence of __GFP_DIRECT_RECLAIM, but I don't see why not
> let kswapd do it.
> 
> Regards.
> Cascardo.
> 
>> The only caveat is that direct reclaim knows which migratetype it wants
>> to allocate and thus changes the highatomic pageblock to. Kswapd doesn't
>> track that, so use MIGRATE_UNMOVABLE. Highatomic allocations usually are
>> not movable, and this will avoid creating nominally movable pageblocks
>> polluted with unmovable pages, so it should be the least problematic
>> default.
>> 
>> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>> ---
>> We interrupt the regular stream of AI fixes by something a bit
>> different. RFC to get agreement on the approach. Haven't tried
>> reproducing the issue in mainline, but it seems quite clear to me from
>> the data I've seen.
>> ---
>>  mm/page_alloc.c | 126 +++++++++++++++++++++++++++++---------------------------
>>  mm/page_alloc.h |   3 ++
>>  mm/vmscan.c     |  16 ++++++-
>>  3 files changed, 83 insertions(+), 62 deletions(-)
>> 
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index a8cac5e01066..785a13570788 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -3476,6 +3476,68 @@ static void reserve_highatomic_pageblock(struct page *page, int order,
>>  	}
>>  }
>>  
>> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
>> +					 bool force)
>> +{
>> +	/*
>> +	 * Preserve at least one pageblock unless memory pressure
>> +	 * is really high.
>> +	 */
>> +	if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages)
>> +		return false;
>> +
>> +	guard(spinlock_irqsave)(&zone->lock);
>> +	for (int order = 0; order < NR_PAGE_ORDERS; order++) {
>> +		struct free_area *area = &(zone->free_area[order]);
>> +		unsigned long size;
>> +		struct page *page;
>> +		int ret;
>> +
>> +		page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
>> +		if (!page)
>> +			continue;
>> +
>> +		size = max(pageblock_nr_pages, 1UL << order);
>> +		/*
>> +		 * It should never happen but changes to locking could
>> +		 * inadvertently allow a per-cpu drain to add pages to
>> +		 * MIGRATE_HIGHATOMIC while unreserving so be safe and watch for
>> +		 * underflows.
>> +		 */
>> +		if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
>> +			size = zone->nr_reserved_highatomic;
>> +		zone->nr_reserved_highatomic -= size;
>> +
>> +		/*
>> +		 * Convert to migratetype and avoid the normal pageblock
>> +		 * stealing heuristics. Minimally, the caller is doing the work
>> +		 * and needs the pages. More importantly, if the block was
>> +		 * always converted to MIGRATE_UNMOVABLE or another type then
>> +		 * the number of pageblocks that cannot be completely freed may
>> +		 * increase.
>> +		 */
>> +		if (order < pageblock_order)
>> +			ret = move_freepages_block(zone, page,
>> +					MIGRATE_HIGHATOMIC, migratetype);
>> +		else {
>> +			move_to_free_list(page, zone, order,
>> +					  MIGRATE_HIGHATOMIC,
>> +					  migratetype);
>> +			change_pageblock_range(page, order, migratetype);
>> +			ret = 1;
>> +		}
>> +		/*
>> +		 * Reserving the block(s) already succeeded,
>> +		 * so this should not fail on zone boundaries.
>> +		 */
>> +		WARN_ON_ONCE(ret == -1);
>> +		if (ret > 0)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>>  /*
>>   * Used when an allocation is about to fail under memory pressure. This
>>   * potentially hurts the reliability of high-order allocations when under
>> @@ -3491,70 +3553,12 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,
>>  	struct zonelist *zonelist = ac->zonelist;
>>  	struct zoneref *z;
>>  	struct zone *zone;
>> -	struct page *page;
>> -	int order;
>> -	int ret;
>>  
>>  	for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx,
>>  								ac->nodemask) {
>> -		/*
>> -		 * Preserve at least one pageblock unless memory pressure
>> -		 * is really high.
>> -		 */
>> -		if (!force && zone->nr_reserved_highatomic <=
>> -					pageblock_nr_pages)
>> -			continue;
>> -
>> -		guard(spinlock_irqsave)(&zone->lock);
>> -		for (order = 0; order < NR_PAGE_ORDERS; order++) {
>> -			struct free_area *area = &(zone->free_area[order]);
>> -			unsigned long size;
>> -
>> -			page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
>> -			if (!page)
>> -				continue;
>> -
>> -			size = max(pageblock_nr_pages, 1UL << order);
>> -			/*
>> -			 * It should never happen but changes to
>> -			 * locking could inadvertently allow a per-cpu
>> -			 * drain to add pages to MIGRATE_HIGHATOMIC
>> -			 * while unreserving so be safe and watch for
>> -			 * underflows.
>> -			 */
>> -			if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
>> -				size = zone->nr_reserved_highatomic;
>> -			zone->nr_reserved_highatomic -= size;
>> -
>> -			/*
>> -			 * Convert to ac->migratetype and avoid the normal
>> -			 * pageblock stealing heuristics. Minimally, the caller
>> -			 * is doing the work and needs the pages. More
>> -			 * importantly, if the block was always converted to
>> -			 * MIGRATE_UNMOVABLE or another type then the number
>> -			 * of pageblocks that cannot be completely freed
>> -			 * may increase.
>> -			 */
>> -			if (order < pageblock_order)
>> -				ret = move_freepages_block(zone, page,
>> -							   MIGRATE_HIGHATOMIC,
>> -							   ac->migratetype);
>> -			else {
>> -				move_to_free_list(page, zone, order,
>> -						  MIGRATE_HIGHATOMIC,
>> -						  ac->migratetype);
>> -				change_pageblock_range(page, order,
>> -						       ac->migratetype);
>> -				ret = 1;
>> -			}
>> -			/*
>> -			 * Reserving the block(s) already succeeded,
>> -			 * so this should not fail on zone boundaries.
>> -			 */
>> -			WARN_ON_ONCE(ret == -1);
>> -			if (ret > 0)
>> -				return ret;
>> -		}
>> +		if (unreserve_highatomic_pageblock_zone(zone,
>> +							ac->migratetype, force))
>> +			return true;
>>  	}
>>  
>>  	return false;
>> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
>> index e238ee91ed24..9fba6480b18d 100644
>> --- a/mm/page_alloc.h
>> +++ b/mm/page_alloc.h
>> @@ -282,6 +282,9 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)
>>  	return list_empty(&area->free_list[migratetype]);
>>  }
>>  
>> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
>> +					 bool force);
>> +
>>  /* Convert GFP flags to their corresponding migrate type */
>>  #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
>>  #define GFP_MOVABLE_SHIFT 3
>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index 9959ca158ce8..91c622068d99 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
>>  		if (nr_boost_reclaim && !nr_reclaimed)
>>  			break;
>>  
>> -		if (raise_priority || !nr_reclaimed)
>> +		if (raise_priority || !nr_reclaimed) {
>> +			/*
>> +			 * Similarly to when direct reclaim is retrying, try to
>> +			 * unreserve highatomic pageblocks on the node. Convert
>> +			 * them to movable ones as kswapd doesn't have a target
>> +			 * migratetype, and passing unmovable should lead to
>> +			 * least permanent fragmentation.
>> +			 */
>> +			for (int i = 0; i <= highest_zoneidx; i++) {
>> +				struct zone *zone = pgdat->node_zones + i;
>> +
>> +				unreserve_highatomic_pageblock_zone(zone,
>> +						MIGRATE_UNMOVABLE, false);
>> +			}
>>  			sc.priority--;
>> +		}
>>  	} while (sc.priority >= 1);
>>  
>>  	/*
>> 
>> ---
>> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
>> change-id: 20261030-kswapd-highatomic-unreserve-7e6629ec65d0
>> 

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Jose A. Perez de Azpillaga]

On 11/4/26 7:38 PM, Vlastimil Babka (SUSE) wrote:
> Hook this highatomic unreserve to the moment we are increasing kswapd
> reclaim priority, as that is when kswapd is struggling to make progress
> and it should roughly match the case of direct reclaim failure.
> 

isn't raise_priority true after almost every pass at DEF_PRIORITY, stuck
or not?

[snip]

> +		if (raise_priority || !nr_reclaimed) {
> +			/*
> +			 * Similarly to when direct reclaim is retrying, try to
> +			 * unreserve highatomic pageblocks on the node. Convert
> +			 * them to movable ones as kswapd doesn't have a target
> +			 * migratetype, and passing unmovable should lead to
> +			 * least permanent fragmentation.
> +			 */
> +			for (int i = 0; i <= highest_zoneidx; i++) {
> +				struct zone *zone = pgdat->node_zones + i;
> +
> +				unreserve_highatomic_pageblock_zone(zone,
> +						MIGRATE_UNMOVABLE, false);
> +			}
s/movable/unmovable/ in the comment?

-- 
cheers,
jose a. p-a

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Vlastimil Babka (SUSE)]

On 11/4/26 22:56, Jose A. Perez de Azpillaga wrote:
> On 11/4/26 7:38 PM, Vlastimil Babka (SUSE) wrote:
>> Hook this highatomic unreserve to the moment we are increasing kswapd
>> reclaim priority, as that is when kswapd is struggling to make progress
>> and it should roughly match the case of direct reclaim failure.
>> 
> 
> isn't raise_priority true after almost every pass at DEF_PRIORITY, stuck
> or not?

Can you elaborate why? At least at first glance, kswapd_shrink_node() should
return true if it scans the requested amount of pages. So I would assume
that although balance_pgdat() will do multiple iterations to reach high
watermarks (because every round of kswapd_shrink_node() will limit itself to
SWAP_CLUSTER_MAX per zone for sc->nr_to_reclaim) this should not be raising
priority if the rounds are successful?

> [snip]
> 
>> +		if (raise_priority || !nr_reclaimed) {
>> +			/*
>> +			 * Similarly to when direct reclaim is retrying, try to
>> +			 * unreserve highatomic pageblocks on the node. Convert
>> +			 * them to movable ones as kswapd doesn't have a target
>> +			 * migratetype, and passing unmovable should lead to
>> +			 * least permanent fragmentation.
>> +			 */
>> +			for (int i = 0; i <= highest_zoneidx; i++) {
>> +				struct zone *zone = pgdat->node_zones + i;
>> +
>> +				unreserve_highatomic_pageblock_zone(zone,
>> +						MIGRATE_UNMOVABLE, false);
>> +			}
> s/movable/unmovable/ in the comment?
> 

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Jose A. Perez de Azpillaga]

On 11/5/26 9:02 AM, Vlastimil Babka (SUSE) wrote:
> On 11/4/26 22:56, Jose A. Perez de Azpillaga wrote:
>> On 11/4/26 7:38 PM, Vlastimil Babka (SUSE) wrote:
>>> Hook this highatomic unreserve to the moment we are increasing kswapd
>>> reclaim priority, as that is when kswapd is struggling to make progress
>>> and it should roughly match the case of direct reclaim failure.
>>>
>>
>> isn't raise_priority true after almost every pass at DEF_PRIORITY, stuck
>> or not?
> 
> Can you elaborate why? At least at first glance, kswapd_shrink_node() should
> return true if it scans the requested amount of pages. So I would assume
> that although balance_pgdat() will do multiple iterations to reach high
> watermarks (because every round of kswapd_shrink_node() will limit itself to
> SWAP_CLUSTER_MAX per zone for sc->nr_to_reclaim) this should not be raising
> priority if the rounds are successful?
> 


it's max(high_wmark_pages(zone), SWAP_CLUSTER_MAX) per zone, so the sum
of the high watermarks. by default that's at least managed/500, while
DEF_PRIORITY scans lru >> 12, so it can't return true above priority 8
even when every round is successful. and sc.priority-- runs before
pgdat_balanced() is rechecked, so a wakeup that balances the node in one
pass still unreserves a pageblock per zone.

-- 
cheers,
jose a. p-a

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Vlastimil Babka (SUSE)]

On 11/5/26 12:43, Jose A. Perez de Azpillaga wrote:
> On 11/5/26 9:02 AM, Vlastimil Babka (SUSE) wrote:
>> On 11/4/26 22:56, Jose A. Perez de Azpillaga wrote:
>>> On 11/4/26 7:38 PM, Vlastimil Babka (SUSE) wrote:
>>>> Hook this highatomic unreserve to the moment we are increasing kswapd
>>>> reclaim priority, as that is when kswapd is struggling to make progress
>>>> and it should roughly match the case of direct reclaim failure.
>>>>
>>>
>>> isn't raise_priority true after almost every pass at DEF_PRIORITY, stuck
>>> or not?
>> 
>> Can you elaborate why? At least at first glance, kswapd_shrink_node() should
>> return true if it scans the requested amount of pages. So I would assume
>> that although balance_pgdat() will do multiple iterations to reach high
>> watermarks (because every round of kswapd_shrink_node() will limit itself to
>> SWAP_CLUSTER_MAX per zone for sc->nr_to_reclaim) this should not be raising
>> priority if the rounds are successful?
>> 
> 
> 
> it's max(high_wmark_pages(zone), SWAP_CLUSTER_MAX) per zone, so the sum
> of the high watermarks. by default that's at least managed/500, while

Ah thanks, as usual I got the min/max logic wrong, hehe.

> DEF_PRIORITY scans lru >> 12, so it can't return true above priority 8
> even when every round is successful. and sc.priority-- runs before
> pgdat_balanced() is rechecked, so a wakeup that balances the node in one
> pass still unreserves a pageblock per zone.

Yeah in that case it would probably be safer to perform the unreserve only
after reaching the highest (lowest numerically) priority and still failing?
I think it should still be enough to recover from the pathological scenario,
which should be rare (but not impossible) in the first place.

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Jose A. Perez de Azpillaga]

On 11/5/26 12:52 PM, Vlastimil Babka (SUSE) wrote:
> On 11/5/26 12:43, Jose A. Perez de Azpillaga wrote:
>> On 11/5/26 9:02 AM, Vlastimil Babka (SUSE) wrote:
>>> On 11/4/26 22:56, Jose A. Perez de Azpillaga wrote:
>>>> On 11/4/26 7:38 PM, Vlastimil Babka (SUSE) wrote:
>>>>> Hook this highatomic unreserve to the moment we are increasing kswapd
>>>>> reclaim priority, as that is when kswapd is struggling to make progress
>>>>> and it should roughly match the case of direct reclaim failure.
>>>>>
>>>>
>>>> isn't raise_priority true after almost every pass at DEF_PRIORITY, stuck
>>>> or not?
>>>
>>> Can you elaborate why? At least at first glance, kswapd_shrink_node() should
>>> return true if it scans the requested amount of pages. So I would assume
>>> that although balance_pgdat() will do multiple iterations to reach high
>>> watermarks (because every round of kswapd_shrink_node() will limit itself to
>>> SWAP_CLUSTER_MAX per zone for sc->nr_to_reclaim) this should not be raising
>>> priority if the rounds are successful?
>>>
>>
>>
>> it's max(high_wmark_pages(zone), SWAP_CLUSTER_MAX) per zone, so the sum
>> of the high watermarks. by default that's at least managed/500, while
> 
> Ah thanks, as usual I got the min/max logic wrong, hehe.
> 

don't worry :)

>> DEF_PRIORITY scans lru >> 12, so it can't return true above priority 8
>> even when every round is successful. and sc.priority-- runs before
>> pgdat_balanced() is rechecked, so a wakeup that balances the node in one
>> pass still unreserves a pageblock per zone.
> 
> Yeah in that case it would probably be safer to perform the unreserve only
> after reaching the highest (lowest numerically) priority and still failing?
> I think it should still be enough to recover from the pathological scenario,
> which should be rare (but not impossible) in the first place.

makes sense to me, as long as kswapd reclaims something. if it doesn't
it goes hopeless after MAX_RECLAIM_RETRIES runs and stays that way until
the node is balanced, so that's only ~16 unreserves per zone.

-- 
cheers,
jose a. p-a

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] mm, vmscan: premature highatomic unreserve in balance_pgdat()
- [Medium] mm, vmscan: incomplete highatomic unreserve in balance_pgdat()
- [Low] mm, vmscan: contradictory migratetype comment in balance_pgdat()
--

commit 9f34d95b6ce713649b203e410b246f2a907d1814
Author: Vlastimil Babka (SUSE) <vbabka@kernel.org>

mm, vmscan: make kswapd unreserve highatomic pageblocks

This commit extracts unreserve_highatomic_pageblock_zone() and calls it from
balance_pgdat() when kswapd raises reclaim priority or reclaims no pages. This
allows kswapd to unreserve MIGRATE_HIGHATOMIC pageblocks on nodes where direct
reclaim does not run.

> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 9959ca158ce83..91c622068d991 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
>  		if (nr_boost_reclaim && !nr_reclaimed)
>  			break;
>  
> -		if (raise_priority || !nr_reclaimed)
> +		if (raise_priority || !nr_reclaimed) {

[Severity: Medium]
Will gating this on raise_priority cause kswapd to prematurely unreserve
MIGRATE_HIGHATOMIC pageblocks during ordinary, successful reclaim passes at
DEF_PRIORITY?

In balance_pgdat(), raise_priority is initialized to true at the start of
each loop iteration and is only cleared if kswapd_shrink_node() returns true
(or at DEF_PRIORITY - 2 during watermark-boosted reclaim):

mm/vmscan.c:kswapd_shrink_node() {
	sc->nr_to_reclaim = 0;
	for_each_managed_zone_pgdat(zone, pgdat, z, sc->reclaim_idx) {
		sc->nr_to_reclaim += max(high_wmark_pages(zone), SWAP_CLUSTER_MAX);
	}
	...
	return max(sc->nr_scanned, sc->nr_reclaimed - nr_reclaimed) >= sc->nr_to_reclaim;
}

Because sc->nr_to_reclaim is set to the full sum of high_wmark_pages(zone)
rather than the deficit between low and high watermarks, a normal pass at
DEF_PRIORITY that scans lruvec_size >> DEF_PRIORITY pages and reclaims
enough pages to balance the node (or aborts scanning early in
should_abort_scan() under CONFIG_LRU_GEN once WMARK_HIGH + MIN_LRU_BATCH is
met) still has kswapd_shrink_node() return false.

Since pgdat_balanced() is only checked at the top of the do-while loop (and
is already true during watermark-boosted reclaim when nr_boost_reclaim > 0),
a first pass at DEF_PRIORITY that makes progress or has already balanced the
node will still enter this branch, unreserve highatomic pageblocks into
MIGRATE_UNMOVABLE across all eligible zones, and only then notice
pgdat_balanced() is true at the top of the next iteration.

Should highatomic unreservation in balance_pgdat() only happen when
!nr_reclaimed (or after rechecking !pgdat_balanced() when reclaim fails to
make progress)?

> +			/*
> +			 * Similarly to when direct reclaim is retrying, try to
> +			 * unreserve highatomic pageblocks on the node. Convert
> +			 * them to movable ones as kswapd doesn't have a target
> +			 * migratetype, and passing unmovable should lead to
> +			 * least permanent fragmentation.
> +			 */

[Severity: Low]
This isn't a bug, but the comment says "Convert them to movable ones" while
the second half of the sentence and the call below use MIGRATE_UNMOVABLE.
Should that say "Convert them to unmovable ones"?

> +			for (int i = 0; i <= highest_zoneidx; i++) {
> +				struct zone *zone = pgdat->node_zones + i;
> +
> +				unreserve_highatomic_pageblock_zone(zone,
> +						MIGRATE_UNMOVABLE, false);
> +			}

[Severity: Medium]
Can a node filled with unreclaimable 1G hugepages still fail to balance in
balance_pgdat() and hit kswapd_failures exhaustion with this change?

There appear to be three related cases in this path:

First, unreserve_highatomic_pageblock_zone() preserves the last pageblock
when force is false:

mm/page_alloc.c:unreserve_highatomic_pageblock_zone() {
	if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages)
		return false;
}

However, __zone_watermark_unusable_free() still subtracts all of
z->nr_free_highatomic, including free pages in that last reserved pageblock.
Unlike direct reclaim, which follows up with force = true in
should_reclaim_retry(), balance_pgdat() never passes force = true, so if the
single remaining highatomic pageblock keeps usable free pages below the
watermark, pgdat_balanced() will remain false.

Second, reserve_highatomic_pageblock() can reserve up to 1% of
zone_managed_pages(zone), which includes 1G hugetlbfs pages (for example, up
to 320 pageblocks on a 64GB zone). Because
unreserve_highatomic_pageblock_zone() unreserves at most one pageblock per
call, a full balance_pgdat() walk from DEF_PRIORITY down to 1 unreserves at
most 12 pageblocks per zone (or at most 6 under CONFIG_LRU_GEN when
set_initial_priority() starts at DEF_PRIORITY / 2 because reclaimable is 0).

Third, unreserving highatomic pageblocks does not increment sc.nr_reclaimed,
and if the final unreserve at sc.priority == 1 balances the node, the loop
exits before rechecking pgdat_balanced() at the top of the loop:

mm/vmscan.c:balance_pgdat() {
	if (!sc.nr_reclaimed && !boosted) {
		int fail_cnt = atomic_inc_return(&pgdat->kswapd_failures);
		/* kswapd context, low overhead to trace every failure */
		trace_mm_vmscan_kswapd_reclaim_fail(pgdat->node_id, fail_cnt);
	}
}

Once pgdat->kswapd_failures reaches MAX_RECLAIM_RETRIES, kswapd stops even if
unreserving highatomic pageblocks was making progress or more reserved
highatomic pageblocks remain.

Could balance_pgdat() pass force = true at low priority, and avoid
incrementing kswapd_failures (or restart/recheck balance) when
unreserve_highatomic_pageblock_zone() succeeds?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261104-kswapd-highatomic-unreserve-v1-1-38be5106adf5@kernel.org?part=1