[PATCH 0/8] mm: Fix several issues with unaccepted memory

[PATCH 0/8] mm: Fix several issues with unaccepted memory

[Kirill A. Shutemov]

The patchset addresses several issues related to unaccepted memory.

The most severe issue is a kswapd hang, which is fixed by patch 1/8.

Patch 2/8 ensures that __alloc_pages_bulk() will not exhaust all
accepted memory without accepting more.

Patches 3/8-5/8 are preparations for patch 6/8, which fixes
alloc_config_page() on machines with unaccepted memory. This allows, for
example, the allocation of gigantic pages at runtime.

Patches 7/8-8/8 enable the kernel to accept memory up to the promo
watermark.

I believe only the first patch deserves backporting.

Please review. Any feedback is welcome.

Kirill A. Shutemov (8):
  mm: Fix endless reclaim on machines with unaccepted memory
  mm: Accept memory in __alloc_pages_bulk().
  mm: Introduce PageUnaccepted() page type
  mm: Rename accept_page() to accept_page_memory()
  mm: Add a helper to accept page
  mm: page_isolation: Handle unaccepted memory isolation
  mm: Introduce promo_wmark_pages()
  mm: Accept to promo watermark

 include/linux/mmzone.h     |   1 +
 include/linux/page-flags.h |   3 +
 kernel/sched/fair.c        |   2 +-
 mm/internal.h              |   8 +++
 mm/page_alloc.c            | 120 ++++++++++++++++++++++++-------------
 mm/page_isolation.c        |   8 +++
 mm/vmscan.c                |   2 +-
 7 files changed, 100 insertions(+), 44 deletions(-)

-- 
2.43.0

[PATCH 1/8] mm: Fix endless reclaim on machines with unaccepted memory

[Kirill A. Shutemov]

Unaccepted memory is considered unusable free memory, which is not
counted as free on the zone watermark check. This causes
get_page_from_freelist() to accept more memory to hit the high
watermark, but it creates problems in the reclaim path.

The reclaim path encounters a failed zone watermark check and attempts
to reclaim memory. This is usually successful, but if there is little or
no reclaimable memory, it can result in endless reclaim with little to
no progress. This can occur early in the boot process, just after start
of the init process when the only reclaimable memory is the page cache
of the init executable and its libraries.

Make unaccepted memory free from watermark check point of view. This way
unaccepted memory will never be the trigger of memory reclaim.
Accept more memory in the get_page_from_freelist() if needed.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Jianxiong Gao <jxgao@google.com>
Fixes: dcdfdd40fa82 ("mm: Add support for unaccepted memory")
Cc: stable@vger.kernel.org # v6.5+
---
 mm/page_alloc.c | 42 ++++++++++++++++++++----------------------
 1 file changed, 20 insertions(+), 22 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 28f80daf5c04..aa9b1eaa638c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -287,7 +287,7 @@ EXPORT_SYMBOL(nr_online_nodes);
 
 static bool page_contains_unaccepted(struct page *page, unsigned int order);
 static void accept_page(struct page *page, unsigned int order);
-static bool try_to_accept_memory(struct zone *zone, unsigned int order);
+static bool cond_accept_memory(struct zone *zone, unsigned int order);
 static inline bool has_unaccepted_memory(void);
 static bool __free_unaccepted(struct page *page);
 
@@ -3072,9 +3072,6 @@ static inline long __zone_watermark_unusable_free(struct zone *z,
 	if (!(alloc_flags & ALLOC_CMA))
 		unusable_free += zone_page_state(z, NR_FREE_CMA_PAGES);
 #endif
-#ifdef CONFIG_UNACCEPTED_MEMORY
-	unusable_free += zone_page_state(z, NR_UNACCEPTED);
-#endif
 
 	return unusable_free;
 }
@@ -3368,6 +3365,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 			}
 		}
 
+		cond_accept_memory(zone, order);
+
 		/*
 		 * Detect whether the number of free pages is below high
 		 * watermark.  If so, we will decrease pcp->high and free
@@ -3393,10 +3392,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 				       gfp_mask)) {
 			int ret;
 
-			if (has_unaccepted_memory()) {
-				if (try_to_accept_memory(zone, order))
-					goto try_this_zone;
-			}
+			if (cond_accept_memory(zone, order))
+				goto try_this_zone;
 
 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
 			/*
@@ -3450,10 +3447,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 
 			return page;
 		} else {
-			if (has_unaccepted_memory()) {
-				if (try_to_accept_memory(zone, order))
-					goto try_this_zone;
-			}
+			if (cond_accept_memory(zone, order))
+				goto try_this_zone;
 
 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
 			/* Try again if zone has deferred pages */
@@ -6951,9 +6946,6 @@ static bool try_to_accept_memory_one(struct zone *zone)
 	struct page *page;
 	bool last;
 
-	if (list_empty(&zone->unaccepted_pages))
-		return false;
-
 	spin_lock_irqsave(&zone->lock, flags);
 	page = list_first_entry_or_null(&zone->unaccepted_pages,
 					struct page, lru);
@@ -6979,23 +6971,29 @@ static bool try_to_accept_memory_one(struct zone *zone)
 	return true;
 }
 
-static bool try_to_accept_memory(struct zone *zone, unsigned int order)
+static bool cond_accept_memory(struct zone *zone, unsigned int order)
 {
 	long to_accept;
-	int ret = false;
+	bool ret = false;
+
+	if (!has_unaccepted_memory())
+		return false;
+
+	if (list_empty(&zone->unaccepted_pages))
+		return false;
 
 	/* How much to accept to get to high watermark? */
 	to_accept = high_wmark_pages(zone) -
 		    (zone_page_state(zone, NR_FREE_PAGES) -
-		    __zone_watermark_unusable_free(zone, order, 0));
+		    __zone_watermark_unusable_free(zone, order, 0) -
+		    zone_page_state(zone, NR_UNACCEPTED));
 
-	/* Accept at least one page */
-	do {
+	while (to_accept > 0) {
 		if (!try_to_accept_memory_one(zone))
 			break;
 		ret = true;
 		to_accept -= MAX_ORDER_NR_PAGES;
-	} while (to_accept > 0);
+	}
 
 	return ret;
 }
@@ -7038,7 +7036,7 @@ static void accept_page(struct page *page, unsigned int order)
 {
 }
 
-static bool try_to_accept_memory(struct zone *zone, unsigned int order)
+static bool cond_accept_memory(struct zone *zone, unsigned int order)
 {
 	return false;
 }
-- 
2.43.0

Re: [PATCH 1/8] mm: Fix endless reclaim on machines with unaccepted memory

[David Hildenbrand]

On 05.08.24 16:59, Kirill A. Shutemov wrote:
> Unaccepted memory is considered unusable free memory, which is not
> counted as free on the zone watermark check. This causes
> get_page_from_freelist() to accept more memory to hit the high
> watermark, but it creates problems in the reclaim path.
> 
> The reclaim path encounters a failed zone watermark check and attempts
> to reclaim memory. This is usually successful, but if there is little or
> no reclaimable memory, it can result in endless reclaim with little to
> no progress. This can occur early in the boot process, just after start
> of the init process when the only reclaimable memory is the page cache
> of the init executable and its libraries.
> 
> Make unaccepted memory free from watermark check point of view. This way
> unaccepted memory will never be the trigger of memory reclaim.
> Accept more memory in the get_page_from_freelist() if needed.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> Reported-by: Jianxiong Gao <jxgao@google.com>
> Fixes: dcdfdd40fa82 ("mm: Add support for unaccepted memory")
> Cc: stable@vger.kernel.org # v6.5+
> ---

Nothing jumped at me, sounds reasonable

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

[PATCH 2/8] mm: Accept memory in __alloc_pages_bulk().

[Kirill A. Shutemov]

Currently, the kernel only accepts memory in get_page_from_freelist(),
but there is another path that directly takes pages from free lists -
__alloc_page_bulk(). This function can consume all accepted memory and
will resort to __alloc_pages_noprof() if necessary.

Conditionally accepted in __alloc_pages_bulk().

The same issue may arise due to deferred page initialization. Kick the
deferred initialization machinery before abandoning the zone, as the
kernel does in get_page_from_freelist().

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/page_alloc.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index aa9b1eaa638c..90a1f01d5996 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4576,12 +4576,25 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 			goto failed;
 		}
 
+		cond_accept_memory(zone, 0);
+retry_this_zone:
 		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages;
 		if (zone_watermark_fast(zone, 0,  mark,
 				zonelist_zone_idx(ac.preferred_zoneref),
 				alloc_flags, gfp)) {
 			break;
 		}
+
+		if (cond_accept_memory(zone, 0))
+			goto retry_this_zone;
+
+#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
+		/* Try again if zone has deferred pages */
+		if (deferred_pages_enabled()) {
+			if (_deferred_grow_zone(zone, 0))
+				goto retry_this_zone;
+		}
+#endif
 	}
 
 	/*
-- 
2.43.0

Re: [PATCH 2/8] mm: Accept memory in __alloc_pages_bulk().

[Mike Rapoport]

On Mon, Aug 05, 2024 at 05:59:34PM +0300, Kirill A. Shutemov wrote:
> Currently, the kernel only accepts memory in get_page_from_freelist(),
> but there is another path that directly takes pages from free lists -
> __alloc_page_bulk(). This function can consume all accepted memory and
> will resort to __alloc_pages_noprof() if necessary.
> 
> Conditionally accepted in __alloc_pages_bulk().
> 
> The same issue may arise due to deferred page initialization. Kick the
> deferred initialization machinery before abandoning the zone, as the
> kernel does in get_page_from_freelist().
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>

> ---
>  mm/page_alloc.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index aa9b1eaa638c..90a1f01d5996 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4576,12 +4576,25 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
>  			goto failed;
>  		}
>  
> +		cond_accept_memory(zone, 0);
> +retry_this_zone:
>  		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages;
>  		if (zone_watermark_fast(zone, 0,  mark,
>  				zonelist_zone_idx(ac.preferred_zoneref),
>  				alloc_flags, gfp)) {
>  			break;
>  		}
> +
> +		if (cond_accept_memory(zone, 0))
> +			goto retry_this_zone;
> +
> +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
> +		/* Try again if zone has deferred pages */
> +		if (deferred_pages_enabled()) {
> +			if (_deferred_grow_zone(zone, 0))
> +				goto retry_this_zone;
> +		}
> +#endif
>  	}
>  
>  	/*
> -- 
> 2.43.0
> 

Re: [PATCH 2/8] mm: Accept memory in __alloc_pages_bulk().

[David Hildenbrand]

On 05.08.24 16:59, Kirill A. Shutemov wrote:
> Currently, the kernel only accepts memory in get_page_from_freelist(),
> but there is another path that directly takes pages from free lists -
> __alloc_page_bulk(). This function can consume all accepted memory and
> will resort to __alloc_pages_noprof() if necessary.
> 
> Conditionally accepted in __alloc_pages_bulk().
> 
> The same issue may arise due to deferred page initialization. Kick the
> deferred initialization machinery before abandoning the zone, as the
> kernel does in get_page_from_freelist().
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>   mm/page_alloc.c | 13 +++++++++++++
>   1 file changed, 13 insertions(+)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index aa9b1eaa638c..90a1f01d5996 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4576,12 +4576,25 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
>   			goto failed;
>   		}
>   
> +		cond_accept_memory(zone, 0);
> +retry_this_zone:
>   		mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages;
>   		if (zone_watermark_fast(zone, 0,  mark,
>   				zonelist_zone_idx(ac.preferred_zoneref),
>   				alloc_flags, gfp)) {
>   			break;
>   		}
> +
> +		if (cond_accept_memory(zone, 0))
> +			goto retry_this_zone;
> +
> +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
> +		/* Try again if zone has deferred pages */
> +		if (deferred_pages_enabled()) {
> +			if (_deferred_grow_zone(zone, 0))
> +				goto retry_this_zone;
> +		}
> +#endif

We could probably avoid the #ifdef if we add a dummy function for 
_deferred_grow_zone().

Same applies to the other similar users in this file.

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

[PATCH 3/8] mm: Introduce PageUnaccepted() page type

[Kirill A. Shutemov]

The new page type allows physical memory scanners to detect unaccepted
memory and handle it accordingly.

The page type is serialized with zone lock.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 include/linux/page-flags.h | 3 +++
 mm/page_alloc.c            | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 5769fe6e4950..e19eac9c2b5c 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -943,6 +943,7 @@ enum pagetype {
 	PG_hugetlb	= 0x04000000,
 	PG_slab		= 0x02000000,
 	PG_zsmalloc	= 0x01000000,
+	PG_unaccepted	= 0x00800000,
 
 	PAGE_TYPE_BASE	= 0x80000000,
 
@@ -1076,6 +1077,8 @@ FOLIO_TEST_FLAG_FALSE(hugetlb)
 
 PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
 
+PAGE_TYPE_OPS(Unaccepted, unaccepted, unaccepted)
+
 /**
  * PageHuge - Determine if the page belongs to hugetlbfs
  * @page: The page to test.
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 90a1f01d5996..a35efb114496 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6972,6 +6972,7 @@ static bool try_to_accept_memory_one(struct zone *zone)
 
 	account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
 	__mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES);
+	__ClearPageUnaccepted(page);
 	spin_unlock_irqrestore(&zone->lock, flags);
 
 	accept_page(page, MAX_PAGE_ORDER);
@@ -7030,6 +7031,7 @@ static bool __free_unaccepted(struct page *page)
 	list_add_tail(&page->lru, &zone->unaccepted_pages);
 	account_freepages(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
 	__mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES);
+	__SetPageUnaccepted(page);
 	spin_unlock_irqrestore(&zone->lock, flags);
 
 	if (first)
-- 
2.43.0

Re: [PATCH 3/8] mm: Introduce PageUnaccepted() page type

[David Hildenbrand]

On 05.08.24 16:59, Kirill A. Shutemov wrote:
> The new page type allows physical memory scanners to detect unaccepted
> memory and handle it accordingly.
> 
> The page type is serialized with zone lock.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>   include/linux/page-flags.h | 3 +++
>   mm/page_alloc.c            | 2 ++
>   2 files changed, 5 insertions(+)
> 
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index 5769fe6e4950..e19eac9c2b5c 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -943,6 +943,7 @@ enum pagetype {
>   	PG_hugetlb	= 0x04000000,
>   	PG_slab		= 0x02000000,
>   	PG_zsmalloc	= 0x01000000,
> +	PG_unaccepted	= 0x00800000,
>   
>   	PAGE_TYPE_BASE	= 0x80000000,
>   
> @@ -1076,6 +1077,8 @@ FOLIO_TEST_FLAG_FALSE(hugetlb)
>   
>   PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
>   
> +PAGE_TYPE_OPS(Unaccepted, unaccepted, unaccepted)

I'm sure you're able to come up with some documentation ;)

> +
>   /**
>    * PageHuge - Determine if the page belongs to hugetlbfs
>    * @page: The page to test.
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 90a1f01d5996..a35efb114496 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -6972,6 +6972,7 @@ static bool try_to_accept_memory_one(struct zone *zone)
>   
>   	account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
>   	__mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES);
> +	__ClearPageUnaccepted(page);
>   	spin_unlock_irqrestore(&zone->lock, flags);
>   
>   	accept_page(page, MAX_PAGE_ORDER);
> @@ -7030,6 +7031,7 @@ static bool __free_unaccepted(struct page *page)
>   	list_add_tail(&page->lru, &zone->unaccepted_pages);
>   	account_freepages(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
>   	__mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES);
> +	__SetPageUnaccepted(page);
>   	spin_unlock_irqrestore(&zone->lock, flags);
>   
>   	if (first)

At the point PG_unaccepted is set/cleared, we don't have another type 
set, right? (IOW, PG_buddy is only set after we cleared PG_unaccepted)

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

Re: [PATCH 3/8] mm: Introduce PageUnaccepted() page type

[Kirill A. Shutemov]

On Tue, Aug 06, 2024 at 02:06:02PM +0200, David Hildenbrand wrote:
> On 05.08.24 16:59, Kirill A. Shutemov wrote:
> > The new page type allows physical memory scanners to detect unaccepted
> > memory and handle it accordingly.
> > 
> > The page type is serialized with zone lock.
> > 
> > Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> > ---
> >   include/linux/page-flags.h | 3 +++
> >   mm/page_alloc.c            | 2 ++
> >   2 files changed, 5 insertions(+)
> > 
> > diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> > index 5769fe6e4950..e19eac9c2b5c 100644
> > --- a/include/linux/page-flags.h
> > +++ b/include/linux/page-flags.h
> > @@ -943,6 +943,7 @@ enum pagetype {
> >   	PG_hugetlb	= 0x04000000,
> >   	PG_slab		= 0x02000000,
> >   	PG_zsmalloc	= 0x01000000,
> > +	PG_unaccepted	= 0x00800000,
> >   	PAGE_TYPE_BASE	= 0x80000000,
> > @@ -1076,6 +1077,8 @@ FOLIO_TEST_FLAG_FALSE(hugetlb)
> >   PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
> > +PAGE_TYPE_OPS(Unaccepted, unaccepted, unaccepted)
> 
> I'm sure you're able to come up with some documentation ;)

Will do.

> > +
> >   /**
> >    * PageHuge - Determine if the page belongs to hugetlbfs
> >    * @page: The page to test.
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index 90a1f01d5996..a35efb114496 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -6972,6 +6972,7 @@ static bool try_to_accept_memory_one(struct zone *zone)
> >   	account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
> >   	__mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES);
> > +	__ClearPageUnaccepted(page);
> >   	spin_unlock_irqrestore(&zone->lock, flags);
> >   	accept_page(page, MAX_PAGE_ORDER);
> > @@ -7030,6 +7031,7 @@ static bool __free_unaccepted(struct page *page)
> >   	list_add_tail(&page->lru, &zone->unaccepted_pages);
> >   	account_freepages(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
> >   	__mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES);
> > +	__SetPageUnaccepted(page);
> >   	spin_unlock_irqrestore(&zone->lock, flags);
> >   	if (first)
> 
> At the point PG_unaccepted is set/cleared, we don't have another type set,
> right? (IOW, PG_buddy is only set after we cleared PG_unaccepted)

Right. PG_buddy is set after we clear PG_unaccepted.

There's brief period when the page is under accept when PG_unaccepted
already cleared, but PG_buddy is not yet set. But I don't think it can be
problematic.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

[PATCH 4/8] mm: Rename accept_page() to accept_page_memory()

[Kirill A. Shutemov]

Rename the helper. The accept_page() name is going to be used for
different function.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/page_alloc.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a35efb114496..34718852d576 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -286,7 +286,7 @@ EXPORT_SYMBOL(nr_online_nodes);
 #endif
 
 static bool page_contains_unaccepted(struct page *page, unsigned int order);
-static void accept_page(struct page *page, unsigned int order);
+static void accept_page_memory(struct page *page, unsigned int order);
 static bool cond_accept_memory(struct zone *zone, unsigned int order);
 static inline bool has_unaccepted_memory(void);
 static bool __free_unaccepted(struct page *page);
@@ -1263,7 +1263,7 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 		if (order == MAX_PAGE_ORDER && __free_unaccepted(page))
 			return;
 
-		accept_page(page, order);
+		accept_page_memory(page, order);
 	}
 
 	/*
@@ -6946,7 +6946,7 @@ static bool page_contains_unaccepted(struct page *page, unsigned int order)
 	return range_contains_unaccepted_memory(start, end);
 }
 
-static void accept_page(struct page *page, unsigned int order)
+static void accept_page_memory(struct page *page, unsigned int order)
 {
 	phys_addr_t start = page_to_phys(page);
 
@@ -6975,7 +6975,7 @@ static bool try_to_accept_memory_one(struct zone *zone)
 	__ClearPageUnaccepted(page);
 	spin_unlock_irqrestore(&zone->lock, flags);
 
-	accept_page(page, MAX_PAGE_ORDER);
+	accept_page_memory(page, MAX_PAGE_ORDER);
 
 	__free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL);
 
@@ -7047,7 +7047,7 @@ static bool page_contains_unaccepted(struct page *page, unsigned int order)
 	return false;
 }
 
-static void accept_page(struct page *page, unsigned int order)
+static void accept_page_memory(struct page *page, unsigned int order)
 {
 }
 
-- 
2.43.0

Re: [PATCH 4/8] mm: Rename accept_page() to accept_page_memory()

[David Hildenbrand]

On 05.08.24 16:59, Kirill A. Shutemov wrote:
> Rename the helper. The accept_page() name is going to be used for
> different function.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>   mm/page_alloc.c | 10 +++++-----
>   1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a35efb114496..34718852d576 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -286,7 +286,7 @@ EXPORT_SYMBOL(nr_online_nodes);
>   #endif
>   
>   static bool page_contains_unaccepted(struct page *page, unsigned int order);
> -static void accept_page(struct page *page, unsigned int order);
> +static void accept_page_memory(struct page *page, unsigned int order);
>   static bool cond_accept_memory(struct zone *zone, unsigned int order);
>   static inline bool has_unaccepted_memory(void);
>   static bool __free_unaccepted(struct page *page);
> @@ -1263,7 +1263,7 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
>   		if (order == MAX_PAGE_ORDER && __free_unaccepted(page))
>   			return;
>   
> -		accept_page(page, order);
> +		accept_page_memory(page, order);

I wonder if we can do better at naming.

Naming them accept_page1 and accept_page2 might be just as confusing as what you use here.

And I think we better just don't have this "page" wrapper here at all and just move to
memory ranges.

After all, the page contains no information we need here besides the PFN.


What about:

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 7a8bdfa742e1..5dc4066f35b3 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -286,7 +286,6 @@ EXPORT_SYMBOL(nr_online_nodes);
  #endif
  
  static bool page_contains_unaccepted(struct page *page, unsigned int order);
-static void accept_page(struct page *page, unsigned int order);
  static inline bool has_unaccepted_memory(void);
  static bool __free_unaccepted(struct page *page);
  
@@ -1262,7 +1261,7 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
                 if (order == MAX_PAGE_ORDER && __free_unaccepted(page))
                         return;
  
-               accept_page(page, order);
+               __accept_memory(page_to_phys(page), PAGE_SIZE << order);
         }
  
         /*
@@ -6975,11 +6974,9 @@ static bool page_contains_unaccepted(struct page *page, unsigned int order)
         return range_contains_unaccepted_memory(start, end);
  }
  
-static void accept_page(struct page *page, unsigned int order)
+static void __accept_memory(phys_addr_t start, phys_addr_t size)
  {
-       phys_addr_t start = page_to_phys(page);
-
-       accept_memory(start, start + (PAGE_SIZE << order));
+       accept_memory(start, start + size);
  }
  
  static bool try_to_accept_memory_one(struct zone *zone)
@@ -7006,7 +7003,7 @@ static bool try_to_accept_memory_one(struct zone *zone)
         __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES);
         spin_unlock_irqrestore(&zone->lock, flags);
  
-       accept_page(page, MAX_PAGE_ORDER);
+       __accept_memory(page_to_phys(page), PAGE_SIZE << MAX_PAGE_ORDER);
  
         __free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL);
  
@@ -7071,7 +7068,7 @@ static bool page_contains_unaccepted(struct page *page, unsigned int order)
         return false;
  }
  
-static void accept_page(struct page *page, unsigned int order)
+static void __accept_memory(phys_addr_t start, phys_addr_t size)
  {
  }
  

It would be even easier if accept_memory() would just accept start+size.

-- 
Cheers,

David / dhildenb

[PATCH 5/8] mm: Add a helper to accept page

[Kirill A. Shutemov]

Accept a given struct page and add it free list.

The help is useful for physical memory scanners that want to use free
unaccepted memory.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/internal.h   |  8 ++++++++
 mm/page_alloc.c | 53 +++++++++++++++++++++++++++++++++++--------------
 2 files changed, 46 insertions(+), 15 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index b4d86436565b..d358535f8a7e 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1578,4 +1578,12 @@ void unlink_file_vma_batch_init(struct unlink_vma_file_batch *);
 void unlink_file_vma_batch_add(struct unlink_vma_file_batch *, struct vm_area_struct *);
 void unlink_file_vma_batch_final(struct unlink_vma_file_batch *);
 
+#ifdef CONFIG_UNACCEPTED_MEMORY
+void accept_page(struct page *page);
+#else /* CONFIG_UNACCEPTED_MEMORY */
+static inline void accept_page(struct page *page)
+{
+}
+#endif /* CONFIG_UNACCEPTED_MEMORY */
+
 #endif	/* __MM_INTERNAL_H */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 34718852d576..5f33e92b0a55 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6953,11 +6953,46 @@ static void accept_page_memory(struct page *page, unsigned int order)
 	accept_memory(start, start + (PAGE_SIZE << order));
 }
 
+static void __accept_page(struct zone *zone, unsigned long *flags,
+			  struct page *page)
+{
+	bool last;
+
+	list_del(&page->lru);
+	last = list_empty(&zone->unaccepted_pages);
+
+	account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
+	__mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES);
+	__ClearPageUnaccepted(page);
+	spin_unlock_irqrestore(&zone->lock, *flags);
+
+	accept_page_memory(page, MAX_PAGE_ORDER);
+
+	__free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL);
+
+	if (last)
+		static_branch_dec(&zones_with_unaccepted_pages);
+}
+
+void accept_page(struct page *page)
+{
+	struct zone *zone = page_zone(page);
+	unsigned long flags;
+
+	spin_lock_irqsave(&zone->lock, flags);
+	if (!PageUnaccepted(page)) {
+		spin_unlock_irqrestore(&zone->lock, flags);
+		return;
+	}
+
+	/* Unlocks zone->lock */
+	__accept_page(zone, &flags, page);
+}
+
 static bool try_to_accept_memory_one(struct zone *zone)
 {
 	unsigned long flags;
 	struct page *page;
-	bool last;
 
 	spin_lock_irqsave(&zone->lock, flags);
 	page = list_first_entry_or_null(&zone->unaccepted_pages,
@@ -6967,20 +7002,8 @@ static bool try_to_accept_memory_one(struct zone *zone)
 		return false;
 	}
 
-	list_del(&page->lru);
-	last = list_empty(&zone->unaccepted_pages);
-
-	account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE);
-	__mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES);
-	__ClearPageUnaccepted(page);
-	spin_unlock_irqrestore(&zone->lock, flags);
-
-	accept_page_memory(page, MAX_PAGE_ORDER);
-
-	__free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL);
-
-	if (last)
-		static_branch_dec(&zones_with_unaccepted_pages);
+	/* Unlocks zone->lock */
+	__accept_page(zone, &flags, page);
 
 	return true;
 }
-- 
2.43.0

Re: [PATCH 5/8] mm: Add a helper to accept page

[David Hildenbrand]

On 05.08.24 16:59, Kirill A. Shutemov wrote:
> Accept a given struct page and add it free list.
> 
> The help is useful for physical memory scanners that want to use free
> unaccepted memory.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

[PATCH 6/8] mm: page_isolation: Handle unaccepted memory isolation

[Kirill A. Shutemov]

Page isolation machinery doesn't know anything about unaccepted memory
and considers it non-free. It leads to alloc_contig_pages() failure.

Treat unaccepted memory as free and accept memory on pageblock
isolation. Once memory is accepted it becomes PageBuddy() and page
isolation knows how to deal with them.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/page_isolation.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 042937d5abe4..39fb8c07aeb7 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -152,6 +152,9 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_
 	unsigned long flags;
 	unsigned long check_unmovable_start, check_unmovable_end;
 
+	if (PageUnaccepted(page))
+		accept_page(page);
+
 	spin_lock_irqsave(&zone->lock, flags);
 
 	/*
@@ -367,6 +370,11 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags,
 		VM_BUG_ON(!page);
 		pfn = page_to_pfn(page);
 
+		if (PageUnaccepted(page)) {
+			pfn += MAX_ORDER_NR_PAGES;
+			continue;
+		}
+
 		if (PageBuddy(page)) {
 			int order = buddy_order(page);
 
-- 
2.43.0

[PATCH 7/8] mm: Introduce promo_wmark_pages()

[Kirill A. Shutemov]

Add promo_wmark_pages() helper to complement other zone watermark
accessors.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 include/linux/mmzone.h | 1 +
 kernel/sched/fair.c    | 2 +-
 mm/vmscan.c            | 2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 41458892bc8a..eb4ec70f6d53 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -671,6 +671,7 @@ enum zone_watermarks {
 #define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost)
 #define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost)
 #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)
+#define promo_wmark_pages(z) (z->_watermark[WMARK_PROMO] + z->watermark_boost)
 #define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost)
 
 /*
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9057584ec06d..6b2a032ba0c6 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1742,7 +1742,7 @@ static bool pgdat_free_space_enough(struct pglist_data *pgdat)
 			continue;
 
 		if (zone_watermark_ok(zone, 0,
-				      wmark_pages(zone, WMARK_PROMO) + enough_wmark,
+				      promo_wmark_pages(zone) + enough_wmark,
 				      ZONE_MOVABLE, 0))
 			return true;
 	}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index cfa839284b92..ad6473eaec28 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6664,7 +6664,7 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
 			continue;
 
 		if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
-			mark = wmark_pages(zone, WMARK_PROMO);
+			mark = promo_wmark_pages(zone);
 		else
 			mark = high_wmark_pages(zone);
 		if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx))
-- 
2.43.0

Re: [PATCH 7/8] mm: Introduce promo_wmark_pages()

[Johannes Weiner]

Hello Kirill,

On Mon, Aug 05, 2024 at 05:59:39PM +0300, Kirill A. Shutemov wrote:
> Add promo_wmark_pages() helper to complement other zone watermark
> accessors.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

Andrew picked up a change that does this a few days ago:

https://lore.kernel.org/all/20240801232548.36604-2-kaiyang2@cs.cmu.edu/T/#u

Re: [PATCH 7/8] mm: Introduce promo_wmark_pages()

[Kirill A. Shutemov]

On Mon, Aug 05, 2024 at 01:04:42PM -0400, Johannes Weiner wrote:
> Hello Kirill,
> 
> On Mon, Aug 05, 2024 at 05:59:39PM +0300, Kirill A. Shutemov wrote:
> > Add promo_wmark_pages() helper to complement other zone watermark
> > accessors.
> > 
> > Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> 
> Andrew picked up a change that does this a few days ago:
> 
> https://lore.kernel.org/all/20240801232548.36604-2-kaiyang2@cs.cmu.edu/T/#u

Ah. Great. I'll drop the patch.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

[PATCH 8/8] mm: Accept to promo watermark

[Kirill A. Shutemov]

Commit c574bbe91703 ("NUMA balancing: optimize page placement for memory
tiering system") introduced a new watermark above "high" -- "promo".

Accept memory memory to the highest watermark which is WMARK_PROMO now.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/page_alloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5f33e92b0a55..fcb15d0e7549 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7019,8 +7019,8 @@ static bool cond_accept_memory(struct zone *zone, unsigned int order)
 	if (list_empty(&zone->unaccepted_pages))
 		return false;
 
-	/* How much to accept to get to high watermark? */
-	to_accept = high_wmark_pages(zone) -
+	/* How much to accept to get to promo watermark? */
+	to_accept = promo_wmark_pages(zone) -
 		    (zone_page_state(zone, NR_FREE_PAGES) -
 		    __zone_watermark_unusable_free(zone, order, 0) -
 		    zone_page_state(zone, NR_UNACCEPTED));
-- 
2.43.0