[RFC PATCH v2 08/18] mm/compaction: free the migrated folios that wait for a grace period

["Kiryl Shutsemau (Meta)" <kas@kernel.org>]

A block that compaction has migrated every folio out of becomes free
once those folios are back in the buddy lists.  A migrated folio that
had PG_rcu_free comes back only after a grace period, so compaction
finds the block still in use, migrates more blocks than it needs and
fails more often.  defrag_mode relies on compaction making whole free
blocks, and there it shows up as allocation stalls.

When compaction moves past a block it migrated from and the zone has
pages waiting, free them before it drains the per-cpu lists and checks
whether it is done.  The drain costs an expedited grace period, so
async compaction, which must not block, skips it.

Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/compaction.c | 8 ++++++++
 mm/page_alloc.c | 2 +-
 mm/page_alloc.h | 1 +
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index a049415512c6..2acb87a3446d 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2760,6 +2760,14 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 				block_start_pfn(cc->migrate_pfn, cc->order);
 
 			if (last_migrated_pfn < current_block_start) {
+				/*
+				 * Folios migrated away from lockless readers
+				 * are freed after a grace period.  Async
+				 * compaction does not wait for it.
+				 */
+				if (cc->mode != MIGRATE_ASYNC &&
+				    zone_rcu_free_waiting(cc->zone))
+					rcu_free_drain();
 				lru_add_drain_cpu_zone(cc->zone);
 				/* No more flushing until we migrate again */
 				last_migrated_pfn = 0;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index eea287ab2a01..afd4343ad572 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1457,7 +1457,7 @@ static void rcu_free_defer(struct folio *folio)
  * callers want to allocate the pages next.  A concurrent drain holds the
  * lock until it has freed what it took.
  */
-static void rcu_free_drain(void)
+void rcu_free_drain(void)
 {
 	static DEFINE_MUTEX(lock);
 	LIST_HEAD(list);
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index 764e3f43b3b9..4f5f06adbfdc 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -235,6 +235,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
 	alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))
 void free_frozen_pages(struct page *page, unsigned int order);
 void free_unref_folios(struct folio_batch *fbatch);
+void rcu_free_drain(void);
 void rcu_free_sync(void);
 
 /* Whether @zone has pages waiting for a grace period */
-- 
2.54.0