[PATCH v3 3/3] mm: add anonymous mTHP folios to the deferred split list

[Joanne Koong <joannelkoong@gmail.com>]

Unlike for PMD-sized folios, an anonymous mTHP folio doesn't get added
to the deferred split list at fault or collapse time. As a result, a
fully mapped mTHP folio that is mostly zero-filled doesn't get split by
the deferred split shrinker when the system is under memory pressure.

At Meta we would like to deploy 2M THP=always on arm64 with 64k base
pages, as 2M gives the contpte benefits while the PMD size (512M) is too
big to use. Without underused splitting this causes memory regressions,
as the unused parts of those folios can never be broken down and
reclaimed.

Add anonymous mTHP folios to the deferred split list from
map_anon_folio_pte_nopf(), mirroring what map_anon_folio_pmd_nopf()
already does for PMD-sized folios. This covers both the fault path and
the khugepaged mTHP collapse path. If there is memory pressure, a
zero-filled mTHP can then be split with its zero pages remapped to the
shared zero page and reclaimed.

Only queue mTHP folios of at least 2M for underused splitting. With
thp_underused() scaling khugepaged/max_ptes_none to the folio size, any
folio of order 2 or above could in principle be underused, but for small
folios it is unlikely to be worth it. An order-2 folio on 4k base pages
can free at most four pages and on systems that only enable small mTHP
sizes (eg Android's order-2-only configuration) zero-filled subpages
tend to be few and short-lived.

The minimum is hard-coded rather than a knob, since admins have no good
way of choosing it, and is a size rather than an order since what a
split recovers depend on the folio's size in bytes, and the same order
means very different sizes across different base page sizes. It can be
lowered later if data shows smaller sizes are worth it. PMD-sized folios
are exempt from the minimum in order to maintain backwards compatibility
since on some architectures (s390, with a 1M PMD) the PMD size can be
below 2M.

To minimize overhead on the common order-0 fault path, the
deferred_split_folio() call is guarded by an inline folio_test_large()
check.

Suggested-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 mm/huge_memory.c | 14 ++++++++++++++
 mm/memory.c      |  2 ++
 2 files changed, 16 insertions(+)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index f758cc1e34fb..cf69020e5a6c 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4561,6 +4561,13 @@ static bool thp_underused_never(unsigned long max_ptes_none)
 	return max_ptes_none == HPAGE_PMD_NR - 1;
 }
 
+/*
+ * Smallest mTHP folio that is queued for underused splitting. Kept
+ * conservative, as small folios are unlikely to free enough memory to be
+ * worth queueing and scanning.
+ */
+#define THP_UNDERUSED_MIN_SIZE	SZ_2M
+
 /* partially_mapped=false won't clear PG_partially_mapped folio flag */
 void deferred_split_folio(struct folio *folio, bool partially_mapped)
 {
@@ -4586,6 +4593,13 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
 		 */
 		if (thp_underused_never(khugepaged_max_ptes_none))
 			return;
+		/*
+		 * Keep queueing PMD-sized folios as before, even where the PMD
+		 * size is below the minimum (eg 1M on s390).
+		 */
+		if (folio_size(folio) < THP_UNDERUSED_MIN_SIZE &&
+		    !is_pmd_order(folio_order(folio)))
+			return;
 	}
 
 	/*
diff --git a/mm/memory.c b/mm/memory.c
index 976cd03954e3..0622425649c6 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5501,6 +5501,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
 	folio_add_lru_vma(folio, vma);
 	set_ptes(vma->vm_mm, addr, pte, entry, nr_pages);
 	update_mmu_cache_range(NULL, vma, addr, pte, nr_pages);
+	if (folio_test_large(folio))
+		deferred_split_folio(folio, false);
 }
 
 static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte,
-- 
2.52.0