* [PATCH v2 00/17] mm/huge_memory: clean up folio split and lift swapcache split limits
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
This series clean up the split code, add better swap cache split support
for mappingless, large order, uniform and non-uniform split. Generic
performance is on par or slightly better, and stack usage is reduced.
The swap cache infrastructure can handle non-uniform or high order folio
replace, so there is no reason for either restriction from the THP side.
What stands in the way is the mixed anon/file folio split routine,
which makes lifting the restrictions hard to follow, and it already
carries some buggy or redundant checks.
So this series cleans up the split path and separates anon and file
splitting into two helpers. The file split path never sees a swap
cache folio, and that is now enforced up front: a folio that is both
in the page cache and the swap cache can only be a shmem folio, which
remains unsupported and is rejected early. That helps to rule out swap
cache handling in that part completely. Only the anon split path
handles swap cache folios, with an anon mapping or mappingless:
either way the splitting is similar, and non-uniform split is
supported as well.
Order-1 is still forbidden for swap cache splitting. In theory it is
doable for shmem swap cache folios, but a mappingless swap cache
folio cannot currently be told apart from a shmem one, so forbid it
for all swap cache folios for now.
Testing:
The in-tree split_huge_page_test selftest (uniform, non-uniform and
in-folio-offset splits of anon and pagecache folios) passes 62/62 on
the patched kernel.
ftrace function_graph tracing filtered on __folio_split() was used to
compare per-call durations between the base and the patched kernel on
the same x86-64 box (interleaved runs across alternating reboots;
mean +- stddev of the per-run averages, 135 split calls per run):
base: 24 runs, 69.6 +- 0.7 us per __folio_split()
patched: 26 runs, 68.8 +- 1.3 us per __folio_split()
The patched kernel is consistently ~1% faster; with this sample
count the difference is outside run-to-run noise.
On x86-64 with gcc 12 (-fstack-usage), the stack frame of
__folio_split() shrinks from 240 to 96 bytes, and the worst-case
split call chain from ~544 to ~384 (anon) or ~464 (file) bytes.
Bloat-o-meter shows a tiny growth of huge_memory.o:
before=58419 after=58446, chg +0.05% (+27 bytes).
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Changes in v2:
- Return -EBUSY instead of -EINVAL for swap cache & shmem folio split
attempt.
- Introduce a for_each_folio_safe macro to dedupliate the code and
hightlight the reason we need to keep the iterate safe from folio
freeing. [ Zi Yan ]
- Rename __split_unmapped_folio() to __split_frozen_folio [ Zi Yan ]
- Rename __folio_freeze_split_unmap_anon. [ Zi Yan ]
- Several comment improments [ Zi Yan ]
- Drop an unused do_lru argument.
- Previouse test results are basically unchanged, stack usage reduced,
object very slightly larger.
- Link to v1: https://patch.msgid.link/20260808-swap-thp-cleanup-v1-0-689939a7ccc3@tencent.com
To: Andrew Morton <akpm@linux-foundation.org>
To: Chris Li <chrisl@kernel.org>
To: Kairui Song <kasong@tencent.com>
To: Kemeng Shi <shikemeng@huaweicloud.com>
To: Nhat Pham <nphamcs@gmail.com>
To: Baoquan He <baoquan.he@linux.dev>
To: Barry Song <baohua@kernel.org>
To: Youngjun Park <youngjun.park@lge.com>
To: David Hildenbrand <david@kernel.org>
To: Lorenzo Stoakes <ljs@kernel.org>
To: Zi Yan <ziy@nvidia.com>
To: Baolin Wang <baolin.wang@linux.alibaba.com>
To: "Liam R. Howlett" <liam@infradead.org>
To: Nico Pache <nico.pache@linux.dev>
To: Ryan Roberts <ryan.roberts@arm.com>
To: Dev Jain <dev.jain@arm.com>
To: Lance Yang <lance.yang@linux.dev>
To: Usama Arif <usama.arif@linux.dev>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
---
Kairui Song (17):
mm/swap: fix off-by-one in swap cache replace sanity check
mm/huge_memory: fix rejection of swap cache folios with a mapping
mm/huge_memory: invert folio_ref_freeze() check to reduce indentation
mm/huge_memory: split the routine for splitting anon and file folio
mm/huge_memory: rename __split_unmapped_folio() to __split_frozen_folio()
mm/huge_memory: consolidate irq and locking for folio split
mm/huge_memory: move EOF trimming into the file split helper
mm/huge_memory: move unmap and remap into the split helpers
mm/huge_memory: move anon_vma and filemap management into split helpers
mm/huge_memory: move memcg switch into the file split helper
mm/huge_memory: allow splitting mappingless swap cache folios
mm/huge_memory: add kerneldoc for the split helpers
mm/huge_memory: drop the unused do_lru argument of the file split helper
mm/huge_memory: clean up after-split folio freeing in __folio_split
mm/huge_memory: lift order-0 restriction for swapcache split
mm/huge_memory: clarify supported split orders in comment
mm/huge_memory: count only swap cache refs in anon folio split
mm/huge_memory.c | 636 ++++++++++++++++++++++++++++++-------------------------
mm/swap_state.c | 3 +-
2 files changed, 344 insertions(+), 295 deletions(-)
---
base-commit: 1029098ee3275ea5b78e329ce132262affa2f8cc
change-id: 20260804-swap-thp-cleanup-6ce2be6cf3b8
Best regards,
--
Kairui Song <kasong@tencent.com>
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH v2 00/17] mm/huge_memory: clean up folio split and lift swapcache split limits
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
This series clean up the split code, add better swap cache split support
for mappingless, large order, uniform and non-uniform split. Generic
performance is on par or slightly better, and stack usage is reduced.
The swap cache infrastructure can handle non-uniform or high order folio
replace, so there is no reason for either restriction from the THP side.
What stands in the way is the mixed anon/file folio split routine,
which makes lifting the restrictions hard to follow, and it already
carries some buggy or redundant checks.
So this series cleans up the split path and separates anon and file
splitting into two helpers. The file split path never sees a swap
cache folio, and that is now enforced up front: a folio that is both
in the page cache and the swap cache can only be a shmem folio, which
remains unsupported and is rejected early. That helps to rule out swap
cache handling in that part completely. Only the anon split path
handles swap cache folios, with an anon mapping or mappingless:
either way the splitting is similar, and non-uniform split is
supported as well.
Order-1 is still forbidden for swap cache splitting. In theory it is
doable for shmem swap cache folios, but a mappingless swap cache
folio cannot currently be told apart from a shmem one, so forbid it
for all swap cache folios for now.
Testing:
The in-tree split_huge_page_test selftest (uniform, non-uniform and
in-folio-offset splits of anon and pagecache folios) passes 62/62 on
the patched kernel.
ftrace function_graph tracing filtered on __folio_split() was used to
compare per-call durations between the base and the patched kernel on
the same x86-64 box (interleaved runs across alternating reboots;
mean +- stddev of the per-run averages, 135 split calls per run):
base: 24 runs, 69.6 +- 0.7 us per __folio_split()
patched: 26 runs, 68.8 +- 1.3 us per __folio_split()
The patched kernel is consistently ~1% faster; with this sample
count the difference is outside run-to-run noise.
On x86-64 with gcc 12 (-fstack-usage), the stack frame of
__folio_split() shrinks from 240 to 96 bytes, and the worst-case
split call chain from ~544 to ~384 (anon) or ~464 (file) bytes.
Bloat-o-meter shows a tiny growth of huge_memory.o:
before=58419 after=58446, chg +0.05% (+27 bytes).
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Changes in v2:
- Return -EBUSY instead of -EINVAL for swap cache & shmem folio split
attempt.
- Introduce a for_each_folio_safe macro to dedupliate the code and
hightlight the reason we need to keep the iterate safe from folio
freeing. [ Zi Yan ]
- Rename __split_unmapped_folio() to __split_frozen_folio [ Zi Yan ]
- Rename __folio_freeze_split_unmap_anon. [ Zi Yan ]
- Several comment improments [ Zi Yan ]
- Drop an unused do_lru argument.
- Previouse test results are basically unchanged, stack usage reduced,
object very slightly larger.
- Link to v1: https://patch.msgid.link/20260808-swap-thp-cleanup-v1-0-689939a7ccc3@tencent.com
To: Andrew Morton <akpm@linux-foundation.org>
To: Chris Li <chrisl@kernel.org>
To: Kairui Song <kasong@tencent.com>
To: Kemeng Shi <shikemeng@huaweicloud.com>
To: Nhat Pham <nphamcs@gmail.com>
To: Baoquan He <baoquan.he@linux.dev>
To: Barry Song <baohua@kernel.org>
To: Youngjun Park <youngjun.park@lge.com>
To: David Hildenbrand <david@kernel.org>
To: Lorenzo Stoakes <ljs@kernel.org>
To: Zi Yan <ziy@nvidia.com>
To: Baolin Wang <baolin.wang@linux.alibaba.com>
To: "Liam R. Howlett" <liam@infradead.org>
To: Nico Pache <nico.pache@linux.dev>
To: Ryan Roberts <ryan.roberts@arm.com>
To: Dev Jain <dev.jain@arm.com>
To: Lance Yang <lance.yang@linux.dev>
To: Usama Arif <usama.arif@linux.dev>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
---
Kairui Song (17):
mm/swap: fix off-by-one in swap cache replace sanity check
mm/huge_memory: fix rejection of swap cache folios with a mapping
mm/huge_memory: invert folio_ref_freeze() check to reduce indentation
mm/huge_memory: split the routine for splitting anon and file folio
mm/huge_memory: rename __split_unmapped_folio() to __split_frozen_folio()
mm/huge_memory: consolidate irq and locking for folio split
mm/huge_memory: move EOF trimming into the file split helper
mm/huge_memory: move unmap and remap into the split helpers
mm/huge_memory: move anon_vma and filemap management into split helpers
mm/huge_memory: move memcg switch into the file split helper
mm/huge_memory: allow splitting mappingless swap cache folios
mm/huge_memory: add kerneldoc for the split helpers
mm/huge_memory: drop the unused do_lru argument of the file split helper
mm/huge_memory: clean up after-split folio freeing in __folio_split
mm/huge_memory: lift order-0 restriction for swapcache split
mm/huge_memory: clarify supported split orders in comment
mm/huge_memory: count only swap cache refs in anon folio split
mm/huge_memory.c | 636 ++++++++++++++++++++++++++++++-------------------------
mm/swap_state.c | 3 +-
2 files changed, 344 insertions(+), 295 deletions(-)
---
base-commit: 1029098ee3275ea5b78e329ce132262affa2f8cc
change-id: 20260804-swap-thp-cleanup-6ce2be6cf3b8
Best regards,
--
Kairui Song <kasong@tencent.com>
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH v2 01/17] mm/swap: fix off-by-one in swap cache replace sanity check
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
The DEBUG_VM sanity check in __swap_cache_replace_folio() iterates
the old folio's range with "while (ci_off++ < ci_end)", so the loop
body runs on the already-incremented offset: the first entry is
skipped and one entry past the range is read. For a folio split
that entry belongs to the first after-split folio and was just
repointed by the replacement loop above, so the check would warn
spuriously whenever sub-folio orders differ from the head folio's,
as non-uniform swapcache splits now do.
Use the same do-while pattern as the replacement loop.
Fixes: 8578e0c00dcf ("mm, swap: use the swap table for the swap cache and switch API")
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/swap_state.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 5be825911e64..f1405e5b813e 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -388,8 +388,9 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
folio_order(old) != folio_order(new)) {
ci_off = swp_cluster_offset(old->swap);
ci_end = ci_off + folio_nr_pages(old);
- while (ci_off++ < ci_end)
+ do {
WARN_ON_ONCE(swp_tb_to_folio(__swap_table_get(ci, ci_off)) != old);
+ } while (++ci_off < ci_end);
}
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 01/17] mm/swap: fix off-by-one in swap cache replace sanity check
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
The DEBUG_VM sanity check in __swap_cache_replace_folio() iterates
the old folio's range with "while (ci_off++ < ci_end)", so the loop
body runs on the already-incremented offset: the first entry is
skipped and one entry past the range is read. For a folio split
that entry belongs to the first after-split folio and was just
repointed by the replacement loop above, so the check would warn
spuriously whenever sub-folio orders differ from the head folio's,
as non-uniform swapcache splits now do.
Use the same do-while pattern as the replacement loop.
Fixes: 8578e0c00dcf ("mm, swap: use the swap table for the swap cache and switch API")
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/swap_state.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 5be825911e64..f1405e5b813e 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -388,8 +388,9 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
folio_order(old) != folio_order(new)) {
ci_off = swp_cluster_offset(old->swap);
ci_end = ci_off + folio_nr_pages(old);
- while (ci_off++ < ci_end)
+ do {
WARN_ON_ONCE(swp_tb_to_folio(__swap_table_get(ci, ci_off)) != old);
+ } while (++ci_off < ci_end);
}
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 02/17] mm/huge_memory: fix rejection of swap cache folios with a mapping
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
A folio in the swap cache cannot be split if it has a mapping (shmem).
The split code does a defensive check for this in
__folio_freeze_and_split_unmapped, after the folio ref has been frozen
and the NR_SHMEM_THPS/NR_FILE_THPS counters have been decremented. It
rejects the split and returns -EINVAL without unfreezing the folio or
restoring the counters. That error path is buggy: if it is ever taken,
it leaves the folio frozen and stuck, skews the counters, and fires
the VM_WARN_ON_ONCE_FOLIO for a state that is actually legitimate.
Check for this case up front in folio_check_splittable and return
-EBUSY before any state is modified, so the split routine always backs
out cleanly.
Also fix a bracket style issue that checkpatch.pl keeps complaining
about.
Fixes: 00527733d0dc ("mm/huge_memory: add two new (not yet used) functions for folio_split()")
Fixes: 714b056c8321 ("mm/huge_memory: convert VM_BUG* to VM_WARN* in __folio_split")
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ced400f72d43..a6759a14e057 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3878,6 +3878,9 @@ static int __split_unmapped_folio(struct folio *folio, int new_order,
int folio_check_splittable(struct folio *folio, unsigned int new_order,
enum split_type split_type)
{
+ bool is_anon = folio_test_anon(folio);
+ bool is_swapcache = folio_test_swapcache(folio);
+
VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
/*
* Folios that just got truncated cannot get split. Signal to the
@@ -3886,11 +3889,11 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
* TODO: this will also currently refuse folios without a mapping in the
* swapcache (shmem or to-be-anon folios).
*/
- if (!folio->mapping && !folio_test_anon(folio))
+ if (!folio->mapping && !is_anon)
return -EBUSY;
/* order-1 is not supported for anonymous THP. */
- if (folio_test_anon(folio) && new_order == 1)
+ if (is_anon && new_order == 1)
return -EINVAL;
/*
@@ -3901,9 +3904,8 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
* swapcache folio split. Only uniform split to order-0 can be used
* here.
*/
- if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && folio_test_swapcache(folio)) {
+ if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && is_swapcache)
return -EINVAL;
- }
if (is_huge_zero_folio(folio))
return -EINVAL;
@@ -3911,6 +3913,15 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
if (folio_test_writeback(folio))
return -EBUSY;
+ /*
+ * A non-anon swapcache folio that still has a mapping can only be a
+ * shmem folio under SWAP IO, it's removed from either swap cache or
+ * shmem mapping afterward. There is little benefit in splitting them
+ * hence reject it here up front before touching anything.
+ */
+ if (!is_anon && is_swapcache && folio->mapping)
+ return -EBUSY;
+
return 0;
}
@@ -3983,14 +3994,8 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
}
}
- if (folio_test_swapcache(folio)) {
- if (mapping) {
- VM_WARN_ON_ONCE_FOLIO(mapping, folio);
- return -EINVAL;
- }
-
+ if (folio_test_swapcache(folio))
ci = swap_cluster_get_and_lock(folio);
- }
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
if (do_lru)
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 02/17] mm/huge_memory: fix rejection of swap cache folios with a mapping
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
A folio in the swap cache cannot be split if it has a mapping (shmem).
The split code does a defensive check for this in
__folio_freeze_and_split_unmapped, after the folio ref has been frozen
and the NR_SHMEM_THPS/NR_FILE_THPS counters have been decremented. It
rejects the split and returns -EINVAL without unfreezing the folio or
restoring the counters. That error path is buggy: if it is ever taken,
it leaves the folio frozen and stuck, skews the counters, and fires
the VM_WARN_ON_ONCE_FOLIO for a state that is actually legitimate.
Check for this case up front in folio_check_splittable and return
-EBUSY before any state is modified, so the split routine always backs
out cleanly.
Also fix a bracket style issue that checkpatch.pl keeps complaining
about.
Fixes: 00527733d0dc ("mm/huge_memory: add two new (not yet used) functions for folio_split()")
Fixes: 714b056c8321 ("mm/huge_memory: convert VM_BUG* to VM_WARN* in __folio_split")
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ced400f72d43..a6759a14e057 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3878,6 +3878,9 @@ static int __split_unmapped_folio(struct folio *folio, int new_order,
int folio_check_splittable(struct folio *folio, unsigned int new_order,
enum split_type split_type)
{
+ bool is_anon = folio_test_anon(folio);
+ bool is_swapcache = folio_test_swapcache(folio);
+
VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
/*
* Folios that just got truncated cannot get split. Signal to the
@@ -3886,11 +3889,11 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
* TODO: this will also currently refuse folios without a mapping in the
* swapcache (shmem or to-be-anon folios).
*/
- if (!folio->mapping && !folio_test_anon(folio))
+ if (!folio->mapping && !is_anon)
return -EBUSY;
/* order-1 is not supported for anonymous THP. */
- if (folio_test_anon(folio) && new_order == 1)
+ if (is_anon && new_order == 1)
return -EINVAL;
/*
@@ -3901,9 +3904,8 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
* swapcache folio split. Only uniform split to order-0 can be used
* here.
*/
- if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && folio_test_swapcache(folio)) {
+ if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && is_swapcache)
return -EINVAL;
- }
if (is_huge_zero_folio(folio))
return -EINVAL;
@@ -3911,6 +3913,15 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
if (folio_test_writeback(folio))
return -EBUSY;
+ /*
+ * A non-anon swapcache folio that still has a mapping can only be a
+ * shmem folio under SWAP IO, it's removed from either swap cache or
+ * shmem mapping afterward. There is little benefit in splitting them
+ * hence reject it here up front before touching anything.
+ */
+ if (!is_anon && is_swapcache && folio->mapping)
+ return -EBUSY;
+
return 0;
}
@@ -3983,14 +3994,8 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
}
}
- if (folio_test_swapcache(folio)) {
- if (mapping) {
- VM_WARN_ON_ONCE_FOLIO(mapping, folio);
- return -EINVAL;
- }
-
+ if (folio_test_swapcache(folio))
ci = swap_cluster_get_and_lock(folio);
- }
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
if (do_lru)
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 03/17] mm/huge_memory: invert folio_ref_freeze() check to reduce indentation
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Invert the folio_ref_freeze() success check in
__folio_freeze_and_split_unmapped() to return early on failure, which
removes one level of indentation from the entire success path.
This is a pure refactoring with no functional change. It prepares the
function to be split into separate helpers for anonymous and
file-backed folios in a later patch.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 181 +++++++++++++++++++++++++++----------------------------
1 file changed, 90 insertions(+), 91 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index a6759a14e057..7fb603ac500f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3940,9 +3940,11 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
pgoff_t end, int *nr_shmem_dropped)
{
struct folio *end_folio = folio_next(folio);
+ struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
struct list_lru_one *lru;
+ struct lruvec *lruvec;
bool dequeue_deferred;
int ret = 0;
@@ -3963,122 +3965,119 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
lru = list_lru_lock(&deferred_split_lru,
folio_nid(folio), &memcg);
}
- if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
- struct swap_cluster_info *ci = NULL;
- struct lruvec *lruvec;
+ if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
if (dequeue_deferred) {
- __list_lru_del(&deferred_split_lru, lru,
- &folio->_deferred_list, folio_nid(folio));
- if (folio_test_partially_mapped(folio)) {
- folio_clear_partially_mapped(folio);
- mod_mthp_stat(old_order,
- MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
- }
list_lru_unlock(lru);
rcu_read_unlock();
}
+ return -EAGAIN;
+ }
- if (mapping) {
- int nr = folio_nr_pages(folio);
-
- if (folio_test_pmd_mappable(folio) &&
- new_order < HPAGE_PMD_ORDER) {
- if (folio_test_swapbacked(folio)) {
- lruvec_stat_mod_folio(folio,
- NR_SHMEM_THPS, -nr);
- } else {
- lruvec_stat_mod_folio(folio,
- NR_FILE_THPS, -nr);
- }
- }
+ if (dequeue_deferred) {
+ __list_lru_del(&deferred_split_lru, lru,
+ &folio->_deferred_list, folio_nid(folio));
+ if (folio_test_partially_mapped(folio)) {
+ folio_clear_partially_mapped(folio);
+ mod_mthp_stat(old_order,
+ MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
+ list_lru_unlock(lru);
+ rcu_read_unlock();
+ }
- if (folio_test_swapcache(folio))
- ci = swap_cluster_get_and_lock(folio);
-
- /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
- if (do_lru)
- lruvec = folio_lruvec_lock(folio);
-
- ret = __split_unmapped_folio(folio, new_order, split_at, xas,
- mapping, split_type);
+ if (mapping) {
+ int nr = folio_nr_pages(folio);
- /*
- * Unfreeze after-split folios and put them back to the right
- * list. @folio should be kept frozon until page cache
- * entries are updated with all the other after-split folios
- * to prevent others seeing stale page cache entries.
- * As a result, new_folio starts from the next folio of
- * @folio.
- */
- for (new_folio = folio_next(folio); new_folio != end_folio;
- new_folio = next) {
- unsigned long nr_pages = folio_nr_pages(new_folio);
+ if (folio_test_pmd_mappable(folio) &&
+ new_order < HPAGE_PMD_ORDER) {
+ if (folio_test_swapbacked(folio)) {
+ lruvec_stat_mod_folio(folio,
+ NR_SHMEM_THPS, -nr);
+ } else {
+ lruvec_stat_mod_folio(folio,
+ NR_FILE_THPS, -nr);
+ }
+ }
+ }
- next = folio_next(new_folio);
+ if (folio_test_swapcache(folio))
+ ci = swap_cluster_get_and_lock(folio);
- zone_device_private_split_cb(folio, new_folio);
+ /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
+ if (do_lru)
+ lruvec = folio_lruvec_lock(folio);
- folio_ref_unfreeze(new_folio,
- folio_cache_ref_count(new_folio) + 1);
+ ret = __split_unmapped_folio(folio, new_order, split_at, xas,
+ mapping, split_type);
- if (do_lru)
- lru_add_split_folio(folio, new_folio, lruvec, list);
+ /*
+ * Unfreeze after-split folios and put them back to the right
+ * list. @folio should be kept frozon until page cache
+ * entries are updated with all the other after-split folios
+ * to prevent others seeing stale page cache entries.
+ * As a result, new_folio starts from the next folio of
+ * @folio.
+ */
+ for (new_folio = folio_next(folio); new_folio != end_folio;
+ new_folio = next) {
+ unsigned long nr_pages = folio_nr_pages(new_folio);
- /*
- * Anonymous folio with swap cache.
- * NOTE: shmem in swap cache is not supported yet.
- */
- if (ci) {
- __swap_cache_replace_folio(ci, folio, new_folio);
- continue;
- }
+ next = folio_next(new_folio);
- /* Anonymous folio without swap cache */
- if (!mapping)
- continue;
+ zone_device_private_split_cb(folio, new_folio);
- /* Add the new folio to the page cache. */
- if (new_folio->index < end) {
- __xa_store(&mapping->i_pages, new_folio->index,
- new_folio, 0);
- continue;
- }
+ folio_ref_unfreeze(new_folio,
+ folio_cache_ref_count(new_folio) + 1);
- VM_WARN_ON_ONCE(!nr_shmem_dropped);
- /* Drop folio beyond EOF: ->index >= end */
- if (shmem_mapping(mapping) && nr_shmem_dropped)
- *nr_shmem_dropped += nr_pages;
- else if (folio_test_clear_dirty(new_folio))
- folio_account_cleaned(
- new_folio, inode_to_wb(mapping->host));
- __filemap_remove_folio(new_folio, NULL);
- folio_put_refs(new_folio, nr_pages);
- }
+ if (do_lru)
+ lru_add_split_folio(folio, new_folio, lruvec, list);
- zone_device_private_split_cb(folio, NULL);
/*
- * Unfreeze @folio only after all page cache entries, which
- * used to point to it, have been updated with new folios.
- * Otherwise, a parallel folio_try_get() can grab @folio
- * and its caller can see stale page cache entries.
+ * Anonymous folio with swap cache.
+ * NOTE: shmem in swap cache is not supported yet.
*/
- folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+ if (ci) {
+ __swap_cache_replace_folio(ci, folio, new_folio);
+ continue;
+ }
- if (do_lru)
- lruvec_unlock(lruvec);
+ /* Anonymous folio without swap cache */
+ if (!mapping)
+ continue;
- if (ci)
- swap_cluster_unlock(ci);
- } else {
- if (dequeue_deferred) {
- list_lru_unlock(lru);
- rcu_read_unlock();
+ /* Add the new folio to the page cache. */
+ if (new_folio->index < end) {
+ __xa_store(&mapping->i_pages, new_folio->index,
+ new_folio, 0);
+ continue;
}
- return -EAGAIN;
+
+ VM_WARN_ON_ONCE(!nr_shmem_dropped);
+ /* Drop folio beyond EOF: ->index >= end */
+ if (shmem_mapping(mapping) && nr_shmem_dropped)
+ *nr_shmem_dropped += nr_pages;
+ else if (folio_test_clear_dirty(new_folio))
+ folio_account_cleaned(new_folio,
+ inode_to_wb(mapping->host));
+ __filemap_remove_folio(new_folio, NULL);
+ folio_put_refs(new_folio, nr_pages);
}
+ zone_device_private_split_cb(folio, NULL);
+ /*
+ * Unfreeze @folio only after all page cache entries, which
+ * used to point to it, have been updated with new folios.
+ * Otherwise, a parallel folio_try_get() can grab @folio
+ * and its caller can see stale page cache entries.
+ */
+ folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+
+ if (do_lru)
+ lruvec_unlock(lruvec);
+ if (ci)
+ swap_cluster_unlock(ci);
+
return ret;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 03/17] mm/huge_memory: invert folio_ref_freeze() check to reduce indentation
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Invert the folio_ref_freeze() success check in
__folio_freeze_and_split_unmapped() to return early on failure, which
removes one level of indentation from the entire success path.
This is a pure refactoring with no functional change. It prepares the
function to be split into separate helpers for anonymous and
file-backed folios in a later patch.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 181 +++++++++++++++++++++++++++----------------------------
1 file changed, 90 insertions(+), 91 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index a6759a14e057..7fb603ac500f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3940,9 +3940,11 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
pgoff_t end, int *nr_shmem_dropped)
{
struct folio *end_folio = folio_next(folio);
+ struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
struct list_lru_one *lru;
+ struct lruvec *lruvec;
bool dequeue_deferred;
int ret = 0;
@@ -3963,122 +3965,119 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
lru = list_lru_lock(&deferred_split_lru,
folio_nid(folio), &memcg);
}
- if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
- struct swap_cluster_info *ci = NULL;
- struct lruvec *lruvec;
+ if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
if (dequeue_deferred) {
- __list_lru_del(&deferred_split_lru, lru,
- &folio->_deferred_list, folio_nid(folio));
- if (folio_test_partially_mapped(folio)) {
- folio_clear_partially_mapped(folio);
- mod_mthp_stat(old_order,
- MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
- }
list_lru_unlock(lru);
rcu_read_unlock();
}
+ return -EAGAIN;
+ }
- if (mapping) {
- int nr = folio_nr_pages(folio);
-
- if (folio_test_pmd_mappable(folio) &&
- new_order < HPAGE_PMD_ORDER) {
- if (folio_test_swapbacked(folio)) {
- lruvec_stat_mod_folio(folio,
- NR_SHMEM_THPS, -nr);
- } else {
- lruvec_stat_mod_folio(folio,
- NR_FILE_THPS, -nr);
- }
- }
+ if (dequeue_deferred) {
+ __list_lru_del(&deferred_split_lru, lru,
+ &folio->_deferred_list, folio_nid(folio));
+ if (folio_test_partially_mapped(folio)) {
+ folio_clear_partially_mapped(folio);
+ mod_mthp_stat(old_order,
+ MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
+ list_lru_unlock(lru);
+ rcu_read_unlock();
+ }
- if (folio_test_swapcache(folio))
- ci = swap_cluster_get_and_lock(folio);
-
- /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
- if (do_lru)
- lruvec = folio_lruvec_lock(folio);
-
- ret = __split_unmapped_folio(folio, new_order, split_at, xas,
- mapping, split_type);
+ if (mapping) {
+ int nr = folio_nr_pages(folio);
- /*
- * Unfreeze after-split folios and put them back to the right
- * list. @folio should be kept frozon until page cache
- * entries are updated with all the other after-split folios
- * to prevent others seeing stale page cache entries.
- * As a result, new_folio starts from the next folio of
- * @folio.
- */
- for (new_folio = folio_next(folio); new_folio != end_folio;
- new_folio = next) {
- unsigned long nr_pages = folio_nr_pages(new_folio);
+ if (folio_test_pmd_mappable(folio) &&
+ new_order < HPAGE_PMD_ORDER) {
+ if (folio_test_swapbacked(folio)) {
+ lruvec_stat_mod_folio(folio,
+ NR_SHMEM_THPS, -nr);
+ } else {
+ lruvec_stat_mod_folio(folio,
+ NR_FILE_THPS, -nr);
+ }
+ }
+ }
- next = folio_next(new_folio);
+ if (folio_test_swapcache(folio))
+ ci = swap_cluster_get_and_lock(folio);
- zone_device_private_split_cb(folio, new_folio);
+ /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
+ if (do_lru)
+ lruvec = folio_lruvec_lock(folio);
- folio_ref_unfreeze(new_folio,
- folio_cache_ref_count(new_folio) + 1);
+ ret = __split_unmapped_folio(folio, new_order, split_at, xas,
+ mapping, split_type);
- if (do_lru)
- lru_add_split_folio(folio, new_folio, lruvec, list);
+ /*
+ * Unfreeze after-split folios and put them back to the right
+ * list. @folio should be kept frozon until page cache
+ * entries are updated with all the other after-split folios
+ * to prevent others seeing stale page cache entries.
+ * As a result, new_folio starts from the next folio of
+ * @folio.
+ */
+ for (new_folio = folio_next(folio); new_folio != end_folio;
+ new_folio = next) {
+ unsigned long nr_pages = folio_nr_pages(new_folio);
- /*
- * Anonymous folio with swap cache.
- * NOTE: shmem in swap cache is not supported yet.
- */
- if (ci) {
- __swap_cache_replace_folio(ci, folio, new_folio);
- continue;
- }
+ next = folio_next(new_folio);
- /* Anonymous folio without swap cache */
- if (!mapping)
- continue;
+ zone_device_private_split_cb(folio, new_folio);
- /* Add the new folio to the page cache. */
- if (new_folio->index < end) {
- __xa_store(&mapping->i_pages, new_folio->index,
- new_folio, 0);
- continue;
- }
+ folio_ref_unfreeze(new_folio,
+ folio_cache_ref_count(new_folio) + 1);
- VM_WARN_ON_ONCE(!nr_shmem_dropped);
- /* Drop folio beyond EOF: ->index >= end */
- if (shmem_mapping(mapping) && nr_shmem_dropped)
- *nr_shmem_dropped += nr_pages;
- else if (folio_test_clear_dirty(new_folio))
- folio_account_cleaned(
- new_folio, inode_to_wb(mapping->host));
- __filemap_remove_folio(new_folio, NULL);
- folio_put_refs(new_folio, nr_pages);
- }
+ if (do_lru)
+ lru_add_split_folio(folio, new_folio, lruvec, list);
- zone_device_private_split_cb(folio, NULL);
/*
- * Unfreeze @folio only after all page cache entries, which
- * used to point to it, have been updated with new folios.
- * Otherwise, a parallel folio_try_get() can grab @folio
- * and its caller can see stale page cache entries.
+ * Anonymous folio with swap cache.
+ * NOTE: shmem in swap cache is not supported yet.
*/
- folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+ if (ci) {
+ __swap_cache_replace_folio(ci, folio, new_folio);
+ continue;
+ }
- if (do_lru)
- lruvec_unlock(lruvec);
+ /* Anonymous folio without swap cache */
+ if (!mapping)
+ continue;
- if (ci)
- swap_cluster_unlock(ci);
- } else {
- if (dequeue_deferred) {
- list_lru_unlock(lru);
- rcu_read_unlock();
+ /* Add the new folio to the page cache. */
+ if (new_folio->index < end) {
+ __xa_store(&mapping->i_pages, new_folio->index,
+ new_folio, 0);
+ continue;
}
- return -EAGAIN;
+
+ VM_WARN_ON_ONCE(!nr_shmem_dropped);
+ /* Drop folio beyond EOF: ->index >= end */
+ if (shmem_mapping(mapping) && nr_shmem_dropped)
+ *nr_shmem_dropped += nr_pages;
+ else if (folio_test_clear_dirty(new_folio))
+ folio_account_cleaned(new_folio,
+ inode_to_wb(mapping->host));
+ __filemap_remove_folio(new_folio, NULL);
+ folio_put_refs(new_folio, nr_pages);
}
+ zone_device_private_split_cb(folio, NULL);
+ /*
+ * Unfreeze @folio only after all page cache entries, which
+ * used to point to it, have been updated with new folios.
+ * Otherwise, a parallel folio_try_get() can grab @folio
+ * and its caller can see stale page cache entries.
+ */
+ folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+
+ if (do_lru)
+ lruvec_unlock(lruvec);
+ if (ci)
+ swap_cluster_unlock(ci);
+
return ret;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 04/17] mm/huge_memory: split the routine for splitting anon and file folio
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
No functional change intended. Before adding more logic, split
__folio_freeze_and_split_unmapped() into an anon and a file variant so
each path can evolve independently. The two paths shared little beyond
the folio freeze call, the LRU locking, and the unfreeze skeleton, but
differed in all other per-folio bookkeeping and routines.
While splitting, some cleanups become easy to apply, and helped drop a
few now-redundant checks. Also introduce a folio iteration helper to
avoid a common pitfall of iterating post-split sub-folios: a sub folio
might get freed mid-iteration as pointed out by Zi [1].
Link: https://lore.kernel.org/linux-mm/DKJSFCLP967N.YBR4DNK1NM2N@nvidia.com/ [1]
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 133 +++++++++++++++++++++++++++++++++++--------------------
1 file changed, 85 insertions(+), 48 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7fb603ac500f..7587eeb09e4a 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3634,6 +3634,18 @@ static bool page_range_has_hwpoisoned(struct page *page, long nr_pages)
return false;
}
+/**
+ * for_each_folio_safe - iterate over contiguous folios safe against folio free
+ * @start: the first folio to iterate
+ * @end: sentinel, folio_next() of the last folio to iterate
+ * @sub_folio: struct folio * to use as the loop cursor
+ * @next: struct folio * used as temporary storage
+ */
+#define for_each_folio_safe(start, end, sub_folio, next) \
+ for (sub_folio = (start), next = folio_next(sub_folio); \
+ sub_folio != (end); \
+ sub_folio = next, next = folio_next(next))
+
/*
* It splits @folio into @new_order folios and copies the @folio metadata to
* all the resulting folios.
@@ -3933,11 +3945,9 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
-static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct xa_state *xas,
- struct address_space *mapping, bool do_lru,
- struct list_head *list, enum split_type split_type,
- pgoff_t end, int *nr_shmem_dropped)
+static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int new_order,
+ struct page *split_at, bool do_lru,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct swap_cluster_info *ci = NULL;
@@ -3948,7 +3958,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
bool dequeue_deferred;
int ret = 0;
- VM_WARN_ON_ONCE(!mapping && end);
/*
* If this folio can be on the deferred split queue, lock out
* the shrinker before freezing the ref. If the shrinker sees
@@ -3956,7 +3965,7 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
* lock and must clean up the LRU state - the same dequeue we
* will do below as part of the split.
*/
- dequeue_deferred = folio_test_anon(folio) && old_order > 1;
+ dequeue_deferred = old_order > 1;
if (dequeue_deferred) {
struct mem_cgroup *memcg;
@@ -3986,24 +3995,70 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
rcu_read_unlock();
}
- if (mapping) {
+ if (folio_test_swapcache(folio))
+ ci = swap_cluster_get_and_lock(folio);
+
+ if (do_lru)
+ lruvec = folio_lruvec_lock(folio);
+
+ ret = __split_unmapped_folio(folio, new_order, split_at, NULL,
+ NULL, split_type);
+
+ /*
+ * Unfreeze the post-split folios and put them back to the right
+ * place. Keep the head @folio frozen until the end: sub entries
+ * in swap cache must be updated first, so a concurrent
+ * swap_cache_get_folio() cannot return the head folio for a sub
+ * entry (folio_try_get() will fail on the head @folio until unfreeze).
+ */
+ for_each_folio_safe(folio_next(folio), end_folio, new_folio, next) {
+ zone_device_private_split_cb(folio, new_folio);
+ folio_ref_unfreeze(new_folio,
+ folio_cache_ref_count(new_folio) + 1);
+ if (do_lru)
+ lru_add_split_folio(folio, new_folio, lruvec, list);
+ if (ci)
+ __swap_cache_replace_folio(ci, folio, new_folio);
+ }
+
+ zone_device_private_split_cb(folio, NULL);
+ folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+
+ if (do_lru)
+ lruvec_unlock(lruvec);
+ if (ci)
+ swap_cluster_unlock(ci);
+
+ return ret;
+}
+
+static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int new_order,
+ struct page *split_at, struct xa_state *xas,
+ struct address_space *mapping, bool do_lru,
+ struct list_head *list, enum split_type split_type,
+ pgoff_t end, int *nr_shmem_dropped)
+{
+ struct folio *end_folio = folio_next(folio);
+ struct folio *new_folio, *next;
+ struct lruvec *lruvec;
+ int ret;
+
+ if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1))
+ return -EAGAIN;
+
+ if (folio_test_pmd_mappable(folio) &&
+ new_order < HPAGE_PMD_ORDER) {
int nr = folio_nr_pages(folio);
- if (folio_test_pmd_mappable(folio) &&
- new_order < HPAGE_PMD_ORDER) {
- if (folio_test_swapbacked(folio)) {
- lruvec_stat_mod_folio(folio,
- NR_SHMEM_THPS, -nr);
- } else {
- lruvec_stat_mod_folio(folio,
- NR_FILE_THPS, -nr);
- }
+ if (folio_test_swapbacked(folio)) {
+ lruvec_stat_mod_folio(folio,
+ NR_SHMEM_THPS, -nr);
+ } else {
+ lruvec_stat_mod_folio(folio,
+ NR_FILE_THPS, -nr);
}
}
- if (folio_test_swapcache(folio))
- ci = swap_cluster_get_and_lock(folio);
-
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
if (do_lru)
lruvec = folio_lruvec_lock(folio);
@@ -4013,39 +4068,21 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
/*
* Unfreeze after-split folios and put them back to the right
- * list. @folio should be kept frozon until page cache
+ * list. @folio should be kept frozen until page cache
* entries are updated with all the other after-split folios
* to prevent others seeing stale page cache entries.
* As a result, new_folio starts from the next folio of
* @folio.
*/
- for (new_folio = folio_next(folio); new_folio != end_folio;
- new_folio = next) {
+ for_each_folio_safe(folio_next(folio), end_folio, new_folio, next) {
unsigned long nr_pages = folio_nr_pages(new_folio);
- next = folio_next(new_folio);
-
- zone_device_private_split_cb(folio, new_folio);
-
folio_ref_unfreeze(new_folio,
folio_cache_ref_count(new_folio) + 1);
if (do_lru)
lru_add_split_folio(folio, new_folio, lruvec, list);
- /*
- * Anonymous folio with swap cache.
- * NOTE: shmem in swap cache is not supported yet.
- */
- if (ci) {
- __swap_cache_replace_folio(ci, folio, new_folio);
- continue;
- }
-
- /* Anonymous folio without swap cache */
- if (!mapping)
- continue;
-
/* Add the new folio to the page cache. */
if (new_folio->index < end) {
__xa_store(&mapping->i_pages, new_folio->index,
@@ -4064,7 +4101,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
folio_put_refs(new_folio, nr_pages);
}
- zone_device_private_split_cb(folio, NULL);
/*
* Unfreeze @folio only after all page cache entries, which
* used to point to it, have been updated with new folios.
@@ -4075,8 +4111,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
if (do_lru)
lruvec_unlock(lruvec);
- if (ci)
- swap_cluster_unlock(ci);
return ret;
}
@@ -4230,10 +4264,14 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
ret = -EAGAIN;
goto fail;
}
+ ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
+ true, list, split_type, end,
+ &nr_shmem_dropped);
+ } else {
+ ret = __folio_freeze_split_unmapped_anon(folio, new_order, split_at, true,
+ list, split_type);
}
- ret = __folio_freeze_and_split_unmapped(folio, new_order, split_at, &xas, mapping,
- true, list, split_type, end, &nr_shmem_dropped);
fail:
if (mapping)
xas_unlock(&xas);
@@ -4333,9 +4371,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
return -EAGAIN;
local_irq_disable();
- ret = __folio_freeze_and_split_unmapped(folio, new_order, &folio->page, NULL,
- NULL, false, NULL, SPLIT_TYPE_UNIFORM,
- 0, NULL);
+ ret = __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
+ false, NULL, SPLIT_TYPE_UNIFORM);
local_irq_enable();
return ret;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 04/17] mm/huge_memory: split the routine for splitting anon and file folio
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
No functional change intended. Before adding more logic, split
__folio_freeze_and_split_unmapped() into an anon and a file variant so
each path can evolve independently. The two paths shared little beyond
the folio freeze call, the LRU locking, and the unfreeze skeleton, but
differed in all other per-folio bookkeeping and routines.
While splitting, some cleanups become easy to apply, and helped drop a
few now-redundant checks. Also introduce a folio iteration helper to
avoid a common pitfall of iterating post-split sub-folios: a sub folio
might get freed mid-iteration as pointed out by Zi [1].
Link: https://lore.kernel.org/linux-mm/DKJSFCLP967N.YBR4DNK1NM2N@nvidia.com/ [1]
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 133 +++++++++++++++++++++++++++++++++++--------------------
1 file changed, 85 insertions(+), 48 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7fb603ac500f..7587eeb09e4a 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3634,6 +3634,18 @@ static bool page_range_has_hwpoisoned(struct page *page, long nr_pages)
return false;
}
+/**
+ * for_each_folio_safe - iterate over contiguous folios safe against folio free
+ * @start: the first folio to iterate
+ * @end: sentinel, folio_next() of the last folio to iterate
+ * @sub_folio: struct folio * to use as the loop cursor
+ * @next: struct folio * used as temporary storage
+ */
+#define for_each_folio_safe(start, end, sub_folio, next) \
+ for (sub_folio = (start), next = folio_next(sub_folio); \
+ sub_folio != (end); \
+ sub_folio = next, next = folio_next(next))
+
/*
* It splits @folio into @new_order folios and copies the @folio metadata to
* all the resulting folios.
@@ -3933,11 +3945,9 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
-static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct xa_state *xas,
- struct address_space *mapping, bool do_lru,
- struct list_head *list, enum split_type split_type,
- pgoff_t end, int *nr_shmem_dropped)
+static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int new_order,
+ struct page *split_at, bool do_lru,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct swap_cluster_info *ci = NULL;
@@ -3948,7 +3958,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
bool dequeue_deferred;
int ret = 0;
- VM_WARN_ON_ONCE(!mapping && end);
/*
* If this folio can be on the deferred split queue, lock out
* the shrinker before freezing the ref. If the shrinker sees
@@ -3956,7 +3965,7 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
* lock and must clean up the LRU state - the same dequeue we
* will do below as part of the split.
*/
- dequeue_deferred = folio_test_anon(folio) && old_order > 1;
+ dequeue_deferred = old_order > 1;
if (dequeue_deferred) {
struct mem_cgroup *memcg;
@@ -3986,24 +3995,70 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
rcu_read_unlock();
}
- if (mapping) {
+ if (folio_test_swapcache(folio))
+ ci = swap_cluster_get_and_lock(folio);
+
+ if (do_lru)
+ lruvec = folio_lruvec_lock(folio);
+
+ ret = __split_unmapped_folio(folio, new_order, split_at, NULL,
+ NULL, split_type);
+
+ /*
+ * Unfreeze the post-split folios and put them back to the right
+ * place. Keep the head @folio frozen until the end: sub entries
+ * in swap cache must be updated first, so a concurrent
+ * swap_cache_get_folio() cannot return the head folio for a sub
+ * entry (folio_try_get() will fail on the head @folio until unfreeze).
+ */
+ for_each_folio_safe(folio_next(folio), end_folio, new_folio, next) {
+ zone_device_private_split_cb(folio, new_folio);
+ folio_ref_unfreeze(new_folio,
+ folio_cache_ref_count(new_folio) + 1);
+ if (do_lru)
+ lru_add_split_folio(folio, new_folio, lruvec, list);
+ if (ci)
+ __swap_cache_replace_folio(ci, folio, new_folio);
+ }
+
+ zone_device_private_split_cb(folio, NULL);
+ folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+
+ if (do_lru)
+ lruvec_unlock(lruvec);
+ if (ci)
+ swap_cluster_unlock(ci);
+
+ return ret;
+}
+
+static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int new_order,
+ struct page *split_at, struct xa_state *xas,
+ struct address_space *mapping, bool do_lru,
+ struct list_head *list, enum split_type split_type,
+ pgoff_t end, int *nr_shmem_dropped)
+{
+ struct folio *end_folio = folio_next(folio);
+ struct folio *new_folio, *next;
+ struct lruvec *lruvec;
+ int ret;
+
+ if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1))
+ return -EAGAIN;
+
+ if (folio_test_pmd_mappable(folio) &&
+ new_order < HPAGE_PMD_ORDER) {
int nr = folio_nr_pages(folio);
- if (folio_test_pmd_mappable(folio) &&
- new_order < HPAGE_PMD_ORDER) {
- if (folio_test_swapbacked(folio)) {
- lruvec_stat_mod_folio(folio,
- NR_SHMEM_THPS, -nr);
- } else {
- lruvec_stat_mod_folio(folio,
- NR_FILE_THPS, -nr);
- }
+ if (folio_test_swapbacked(folio)) {
+ lruvec_stat_mod_folio(folio,
+ NR_SHMEM_THPS, -nr);
+ } else {
+ lruvec_stat_mod_folio(folio,
+ NR_FILE_THPS, -nr);
}
}
- if (folio_test_swapcache(folio))
- ci = swap_cluster_get_and_lock(folio);
-
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
if (do_lru)
lruvec = folio_lruvec_lock(folio);
@@ -4013,39 +4068,21 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
/*
* Unfreeze after-split folios and put them back to the right
- * list. @folio should be kept frozon until page cache
+ * list. @folio should be kept frozen until page cache
* entries are updated with all the other after-split folios
* to prevent others seeing stale page cache entries.
* As a result, new_folio starts from the next folio of
* @folio.
*/
- for (new_folio = folio_next(folio); new_folio != end_folio;
- new_folio = next) {
+ for_each_folio_safe(folio_next(folio), end_folio, new_folio, next) {
unsigned long nr_pages = folio_nr_pages(new_folio);
- next = folio_next(new_folio);
-
- zone_device_private_split_cb(folio, new_folio);
-
folio_ref_unfreeze(new_folio,
folio_cache_ref_count(new_folio) + 1);
if (do_lru)
lru_add_split_folio(folio, new_folio, lruvec, list);
- /*
- * Anonymous folio with swap cache.
- * NOTE: shmem in swap cache is not supported yet.
- */
- if (ci) {
- __swap_cache_replace_folio(ci, folio, new_folio);
- continue;
- }
-
- /* Anonymous folio without swap cache */
- if (!mapping)
- continue;
-
/* Add the new folio to the page cache. */
if (new_folio->index < end) {
__xa_store(&mapping->i_pages, new_folio->index,
@@ -4064,7 +4101,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
folio_put_refs(new_folio, nr_pages);
}
- zone_device_private_split_cb(folio, NULL);
/*
* Unfreeze @folio only after all page cache entries, which
* used to point to it, have been updated with new folios.
@@ -4075,8 +4111,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
if (do_lru)
lruvec_unlock(lruvec);
- if (ci)
- swap_cluster_unlock(ci);
return ret;
}
@@ -4230,10 +4264,14 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
ret = -EAGAIN;
goto fail;
}
+ ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
+ true, list, split_type, end,
+ &nr_shmem_dropped);
+ } else {
+ ret = __folio_freeze_split_unmapped_anon(folio, new_order, split_at, true,
+ list, split_type);
}
- ret = __folio_freeze_and_split_unmapped(folio, new_order, split_at, &xas, mapping,
- true, list, split_type, end, &nr_shmem_dropped);
fail:
if (mapping)
xas_unlock(&xas);
@@ -4333,9 +4371,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
return -EAGAIN;
local_irq_disable();
- ret = __folio_freeze_and_split_unmapped(folio, new_order, &folio->page, NULL,
- NULL, false, NULL, SPLIT_TYPE_UNIFORM,
- 0, NULL);
+ ret = __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
+ false, NULL, SPLIT_TYPE_UNIFORM);
local_irq_enable();
return ret;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 05/17] mm/huge_memory: rename __split_unmapped_folio() to __split_frozen_folio()
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
The helper splits a folio whose refcount is frozen: the frozen refcount
is the state it relies on, while unmapping is arranged by the caller
beforehand. The old name caused confusion and people may try to call the
helper on non-frozen folios.
Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7587eeb09e4a..dfecb93dd64f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3767,8 +3767,8 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
}
/**
- * __split_unmapped_folio() - splits an unmapped @folio to lower order folios in
- * two ways: uniform split or non-uniform split.
+ * __split_frozen_folio() - splits a frozen @folio to lower order folios
+ * in two ways: uniform split or non-uniform split.
* @folio: the to-be-split folio
* @new_order: the smallest order of the after split folios (since buddy
* allocator like split generates folios with orders from @folio's
@@ -3807,7 +3807,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
* Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
* split but not to @new_order, the caller needs to check)
*/
-static int __split_unmapped_folio(struct folio *folio, int new_order,
+static int __split_frozen_folio(struct folio *folio, int new_order,
struct page *split_at, struct xa_state *xas,
struct address_space *mapping, enum split_type split_type)
{
@@ -4001,8 +4001,8 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
if (do_lru)
lruvec = folio_lruvec_lock(folio);
- ret = __split_unmapped_folio(folio, new_order, split_at, NULL,
- NULL, split_type);
+ ret = __split_frozen_folio(folio, new_order, split_at, NULL,
+ NULL, split_type);
/*
* Unfreeze the post-split folios and put them back to the right
@@ -4063,8 +4063,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (do_lru)
lruvec = folio_lruvec_lock(folio);
- ret = __split_unmapped_folio(folio, new_order, split_at, xas,
- mapping, split_type);
+ ret = __split_frozen_folio(folio, new_order, split_at, xas,
+ mapping, split_type);
/*
* Unfreeze after-split folios and put them back to the right
@@ -4124,9 +4124,9 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
* @list: after-split folios will be put on it if non NULL
* @split_type: perform uniform split or not (non-uniform split)
*
- * It calls __split_unmapped_folio() to perform uniform and non-uniform split.
+ * It calls __split_frozen_folio() to perform uniform and non-uniform split.
* It is in charge of checking whether the split is supported or not and
- * preparing @folio for __split_unmapped_folio().
+ * preparing @folio for __split_frozen_folio().
*
* After splitting, the after-split folio containing @lock_at remains locked
* and others are unlocked:
@@ -4229,7 +4229,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
i_mmap_lock_read(mapping);
/*
- *__split_unmapped_folio() may need to trim off pages beyond
+ * __split_frozen_folio() may need to trim off pages beyond
* EOF: but on 32-bit, i_size_read() takes an irq-unsafe
* seqlock, which cannot be nested inside the page tree lock.
* So note end now: i_size itself may be changed at any moment,
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 05/17] mm/huge_memory: rename __split_unmapped_folio() to __split_frozen_folio()
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
The helper splits a folio whose refcount is frozen: the frozen refcount
is the state it relies on, while unmapping is arranged by the caller
beforehand. The old name caused confusion and people may try to call the
helper on non-frozen folios.
Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7587eeb09e4a..dfecb93dd64f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3767,8 +3767,8 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
}
/**
- * __split_unmapped_folio() - splits an unmapped @folio to lower order folios in
- * two ways: uniform split or non-uniform split.
+ * __split_frozen_folio() - splits a frozen @folio to lower order folios
+ * in two ways: uniform split or non-uniform split.
* @folio: the to-be-split folio
* @new_order: the smallest order of the after split folios (since buddy
* allocator like split generates folios with orders from @folio's
@@ -3807,7 +3807,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
* Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
* split but not to @new_order, the caller needs to check)
*/
-static int __split_unmapped_folio(struct folio *folio, int new_order,
+static int __split_frozen_folio(struct folio *folio, int new_order,
struct page *split_at, struct xa_state *xas,
struct address_space *mapping, enum split_type split_type)
{
@@ -4001,8 +4001,8 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
if (do_lru)
lruvec = folio_lruvec_lock(folio);
- ret = __split_unmapped_folio(folio, new_order, split_at, NULL,
- NULL, split_type);
+ ret = __split_frozen_folio(folio, new_order, split_at, NULL,
+ NULL, split_type);
/*
* Unfreeze the post-split folios and put them back to the right
@@ -4063,8 +4063,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (do_lru)
lruvec = folio_lruvec_lock(folio);
- ret = __split_unmapped_folio(folio, new_order, split_at, xas,
- mapping, split_type);
+ ret = __split_frozen_folio(folio, new_order, split_at, xas,
+ mapping, split_type);
/*
* Unfreeze after-split folios and put them back to the right
@@ -4124,9 +4124,9 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
* @list: after-split folios will be put on it if non NULL
* @split_type: perform uniform split or not (non-uniform split)
*
- * It calls __split_unmapped_folio() to perform uniform and non-uniform split.
+ * It calls __split_frozen_folio() to perform uniform and non-uniform split.
* It is in charge of checking whether the split is supported or not and
- * preparing @folio for __split_unmapped_folio().
+ * preparing @folio for __split_frozen_folio().
*
* After splitting, the after-split folio containing @lock_at remains locked
* and others are unlocked:
@@ -4229,7 +4229,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
i_mmap_lock_read(mapping);
/*
- *__split_unmapped_folio() may need to trim off pages beyond
+ * __split_frozen_folio() may need to trim off pages beyond
* EOF: but on 32-bit, i_size_read() takes an irq-unsafe
* seqlock, which cannot be nested inside the page tree lock.
* So note end now: i_size itself may be changed at any moment,
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 06/17] mm/huge_memory: consolidate irq and locking for folio split
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Let each split helper handle its own locking instead of relying on
the caller, so both helpers manage their own irq and locking state.
This lets __folio_split() drop its local irq handling and fail label,
preparing for further cleanup.
The file path now uses xas_lock_irq() instead of local_irq_disable()
with xas_lock(). The two are equivalent on non-RT, and
TRANSPARENT_HUGEPAGE cannot be enabled on RT anyway. This conversion
also buys consistency: every other place in mm/ that freezes a folio
while it is still reachable through the page cache already takes the
lock this way. This was actually the last plain xas_lock() on
mapping->i_pages left in mm. If we are going to support RT, spinning
on frozen folio refs could be a problem, but it already exists in
many places and should be fixed generically.
The anon helper keeps a single local_irq_disable() as before, because
it has to cover several plain spinlocks at once.
The dropped xas_reset() was a no-op as the xa_state is not walked
before the xas_load() under the lock.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 52 ++++++++++++++++++++++++----------------------------
1 file changed, 24 insertions(+), 28 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index dfecb93dd64f..2cd53afac63e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3958,6 +3958,8 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
bool dequeue_deferred;
int ret = 0;
+ local_irq_disable();
+
/*
* If this folio can be on the deferred split queue, lock out
* the shrinker before freezing the ref. If the shrinker sees
@@ -3980,6 +3982,7 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
list_lru_unlock(lru);
rcu_read_unlock();
}
+ local_irq_enable();
return -EAGAIN;
}
@@ -4028,6 +4031,7 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
lruvec_unlock(lruvec);
if (ci)
swap_cluster_unlock(ci);
+ local_irq_enable();
return ret;
}
@@ -4043,8 +4047,21 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
struct lruvec *lruvec;
int ret;
- if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1))
- return -EAGAIN;
+ xas_lock_irq(xas);
+
+ /*
+ * Check if the folio is present in page cache.
+ * We assume all tail are present too, if folio is there.
+ */
+ if (xas_load(xas) != folio) {
+ ret = -EAGAIN;
+ goto fail;
+ }
+
+ if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
+ ret = -EAGAIN;
+ goto fail;
+ }
if (folio_test_pmd_mappable(folio) &&
new_order < HPAGE_PMD_ORDER) {
@@ -4112,6 +4129,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (do_lru)
lruvec_unlock(lruvec);
+fail:
+ xas_unlock_irq(xas);
return ret;
}
@@ -4251,19 +4270,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
unmap_folio(folio);
- /* block interrupt reentry in xa_lock and spinlock */
- local_irq_disable();
- if (mapping) {
- /*
- * Check if the folio is present in page cache.
- * We assume all tail are present too, if folio is there.
- */
- xas_lock(&xas);
- xas_reset(&xas);
- if (xas_load(&xas) != folio) {
- ret = -EAGAIN;
- goto fail;
- }
+ if (!is_anon) {
ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
true, list, split_type, end,
&nr_shmem_dropped);
@@ -4272,12 +4279,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
list, split_type);
}
-fail:
- if (mapping)
- xas_unlock(&xas);
-
- local_irq_enable();
-
if (nr_shmem_dropped)
shmem_uncharge(mapping->host, nr_shmem_dropped);
@@ -4360,8 +4361,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
*/
int folio_split_unmapped(struct folio *folio, unsigned int new_order)
{
- int ret = 0;
-
VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
@@ -4370,11 +4369,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
return -EAGAIN;
- local_irq_disable();
- ret = __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
- false, NULL, SPLIT_TYPE_UNIFORM);
- local_irq_enable();
- return ret;
+ return __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
+ false, NULL, SPLIT_TYPE_UNIFORM);
}
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 06/17] mm/huge_memory: consolidate irq and locking for folio split
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Let each split helper handle its own locking instead of relying on
the caller, so both helpers manage their own irq and locking state.
This lets __folio_split() drop its local irq handling and fail label,
preparing for further cleanup.
The file path now uses xas_lock_irq() instead of local_irq_disable()
with xas_lock(). The two are equivalent on non-RT, and
TRANSPARENT_HUGEPAGE cannot be enabled on RT anyway. This conversion
also buys consistency: every other place in mm/ that freezes a folio
while it is still reachable through the page cache already takes the
lock this way. This was actually the last plain xas_lock() on
mapping->i_pages left in mm. If we are going to support RT, spinning
on frozen folio refs could be a problem, but it already exists in
many places and should be fixed generically.
The anon helper keeps a single local_irq_disable() as before, because
it has to cover several plain spinlocks at once.
The dropped xas_reset() was a no-op as the xa_state is not walked
before the xas_load() under the lock.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 52 ++++++++++++++++++++++++----------------------------
1 file changed, 24 insertions(+), 28 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index dfecb93dd64f..2cd53afac63e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3958,6 +3958,8 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
bool dequeue_deferred;
int ret = 0;
+ local_irq_disable();
+
/*
* If this folio can be on the deferred split queue, lock out
* the shrinker before freezing the ref. If the shrinker sees
@@ -3980,6 +3982,7 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
list_lru_unlock(lru);
rcu_read_unlock();
}
+ local_irq_enable();
return -EAGAIN;
}
@@ -4028,6 +4031,7 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
lruvec_unlock(lruvec);
if (ci)
swap_cluster_unlock(ci);
+ local_irq_enable();
return ret;
}
@@ -4043,8 +4047,21 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
struct lruvec *lruvec;
int ret;
- if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1))
- return -EAGAIN;
+ xas_lock_irq(xas);
+
+ /*
+ * Check if the folio is present in page cache.
+ * We assume all tail are present too, if folio is there.
+ */
+ if (xas_load(xas) != folio) {
+ ret = -EAGAIN;
+ goto fail;
+ }
+
+ if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
+ ret = -EAGAIN;
+ goto fail;
+ }
if (folio_test_pmd_mappable(folio) &&
new_order < HPAGE_PMD_ORDER) {
@@ -4112,6 +4129,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (do_lru)
lruvec_unlock(lruvec);
+fail:
+ xas_unlock_irq(xas);
return ret;
}
@@ -4251,19 +4270,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
unmap_folio(folio);
- /* block interrupt reentry in xa_lock and spinlock */
- local_irq_disable();
- if (mapping) {
- /*
- * Check if the folio is present in page cache.
- * We assume all tail are present too, if folio is there.
- */
- xas_lock(&xas);
- xas_reset(&xas);
- if (xas_load(&xas) != folio) {
- ret = -EAGAIN;
- goto fail;
- }
+ if (!is_anon) {
ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
true, list, split_type, end,
&nr_shmem_dropped);
@@ -4272,12 +4279,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
list, split_type);
}
-fail:
- if (mapping)
- xas_unlock(&xas);
-
- local_irq_enable();
-
if (nr_shmem_dropped)
shmem_uncharge(mapping->host, nr_shmem_dropped);
@@ -4360,8 +4361,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
*/
int folio_split_unmapped(struct folio *folio, unsigned int new_order)
{
- int ret = 0;
-
VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
@@ -4370,11 +4369,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
return -EAGAIN;
- local_irq_disable();
- ret = __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
- false, NULL, SPLIT_TYPE_UNIFORM);
- local_irq_enable();
- return ret;
+ return __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
+ false, NULL, SPLIT_TYPE_UNIFORM);
}
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 07/17] mm/huge_memory: move EOF trimming into the file split helper
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Instead of receiving @end and @nr_shmem_dropped from the caller, the
file split helper now computes the EOF boundary and trims pages beyond
it itself, as this is only needed for file split. This drops the
redundant parameter passing and sanity check.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 42 +++++++++++++++++++-----------------------
1 file changed, 19 insertions(+), 23 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 2cd53afac63e..43093b9a5bca 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4039,14 +4039,26 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int new_order,
struct page *split_at, struct xa_state *xas,
struct address_space *mapping, bool do_lru,
- struct list_head *list, enum split_type split_type,
- pgoff_t end, int *nr_shmem_dropped)
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
+ int nr_shmem_dropped = 0;
struct lruvec *lruvec;
+ pgoff_t end = 0;
int ret;
+ /*
+ * __split_frozen_folio() may need to trim off pages beyond
+ * EOF: but on 32-bit, i_size_read() takes an irq-unsafe
+ * seqlock, which cannot be nested inside the page tree lock.
+ * So note end now: i_size itself may be changed at any moment,
+ * but folio lock is good enough to serialize the trimming.
+ */
+ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
+ if (shmem_mapping(mapping))
+ end = shmem_fallocend(mapping->host, end);
+
xas_lock_irq(xas);
/*
@@ -4107,10 +4119,9 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
continue;
}
- VM_WARN_ON_ONCE(!nr_shmem_dropped);
/* Drop folio beyond EOF: ->index >= end */
- if (shmem_mapping(mapping) && nr_shmem_dropped)
- *nr_shmem_dropped += nr_pages;
+ if (shmem_mapping(mapping))
+ nr_shmem_dropped += nr_pages;
else if (folio_test_clear_dirty(new_folio))
folio_account_cleaned(new_folio,
inode_to_wb(mapping->host));
@@ -4131,6 +4142,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
fail:
xas_unlock_irq(xas);
+ if (nr_shmem_dropped)
+ shmem_uncharge(mapping->host, nr_shmem_dropped);
return ret;
}
@@ -4167,9 +4180,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
- int nr_shmem_dropped = 0;
enum ttu_flags ttu_flags = 0;
- pgoff_t end = 0;
int ret;
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
@@ -4246,17 +4257,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
anon_vma = NULL;
i_mmap_lock_read(mapping);
-
- /*
- * __split_frozen_folio() may need to trim off pages beyond
- * EOF: but on 32-bit, i_size_read() takes an irq-unsafe
- * seqlock, which cannot be nested inside the page tree lock.
- * So note end now: i_size itself may be changed at any moment,
- * but folio lock is good enough to serialize the trimming.
- */
- end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
- if (shmem_mapping(mapping))
- end = shmem_fallocend(mapping->host, end);
}
/*
@@ -4272,16 +4272,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (!is_anon) {
ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
- true, list, split_type, end,
- &nr_shmem_dropped);
+ true, list, split_type);
} else {
ret = __folio_freeze_split_unmapped_anon(folio, new_order, split_at, true,
list, split_type);
}
- if (nr_shmem_dropped)
- shmem_uncharge(mapping->host, nr_shmem_dropped);
-
if (!ret && is_anon && !folio_is_device_private(folio))
ttu_flags = TTU_USE_SHARED_ZEROPAGE;
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 07/17] mm/huge_memory: move EOF trimming into the file split helper
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Instead of receiving @end and @nr_shmem_dropped from the caller, the
file split helper now computes the EOF boundary and trims pages beyond
it itself, as this is only needed for file split. This drops the
redundant parameter passing and sanity check.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 42 +++++++++++++++++++-----------------------
1 file changed, 19 insertions(+), 23 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 2cd53afac63e..43093b9a5bca 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4039,14 +4039,26 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int new_order,
struct page *split_at, struct xa_state *xas,
struct address_space *mapping, bool do_lru,
- struct list_head *list, enum split_type split_type,
- pgoff_t end, int *nr_shmem_dropped)
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
+ int nr_shmem_dropped = 0;
struct lruvec *lruvec;
+ pgoff_t end = 0;
int ret;
+ /*
+ * __split_frozen_folio() may need to trim off pages beyond
+ * EOF: but on 32-bit, i_size_read() takes an irq-unsafe
+ * seqlock, which cannot be nested inside the page tree lock.
+ * So note end now: i_size itself may be changed at any moment,
+ * but folio lock is good enough to serialize the trimming.
+ */
+ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
+ if (shmem_mapping(mapping))
+ end = shmem_fallocend(mapping->host, end);
+
xas_lock_irq(xas);
/*
@@ -4107,10 +4119,9 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
continue;
}
- VM_WARN_ON_ONCE(!nr_shmem_dropped);
/* Drop folio beyond EOF: ->index >= end */
- if (shmem_mapping(mapping) && nr_shmem_dropped)
- *nr_shmem_dropped += nr_pages;
+ if (shmem_mapping(mapping))
+ nr_shmem_dropped += nr_pages;
else if (folio_test_clear_dirty(new_folio))
folio_account_cleaned(new_folio,
inode_to_wb(mapping->host));
@@ -4131,6 +4142,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
fail:
xas_unlock_irq(xas);
+ if (nr_shmem_dropped)
+ shmem_uncharge(mapping->host, nr_shmem_dropped);
return ret;
}
@@ -4167,9 +4180,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
- int nr_shmem_dropped = 0;
enum ttu_flags ttu_flags = 0;
- pgoff_t end = 0;
int ret;
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
@@ -4246,17 +4257,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
anon_vma = NULL;
i_mmap_lock_read(mapping);
-
- /*
- * __split_frozen_folio() may need to trim off pages beyond
- * EOF: but on 32-bit, i_size_read() takes an irq-unsafe
- * seqlock, which cannot be nested inside the page tree lock.
- * So note end now: i_size itself may be changed at any moment,
- * but folio lock is good enough to serialize the trimming.
- */
- end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
- if (shmem_mapping(mapping))
- end = shmem_fallocend(mapping->host, end);
}
/*
@@ -4272,16 +4272,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (!is_anon) {
ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
- true, list, split_type, end,
- &nr_shmem_dropped);
+ true, list, split_type);
} else {
ret = __folio_freeze_split_unmapped_anon(folio, new_order, split_at, true,
list, split_type);
}
- if (nr_shmem_dropped)
- shmem_uncharge(mapping->host, nr_shmem_dropped);
-
if (!ret && is_anon && !folio_is_device_private(folio))
ttu_flags = TTU_USE_SHARED_ZEROPAGE;
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 08/17] mm/huge_memory: move unmap and remap into the split helpers
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
To prepare for further cleanup, move the unmap/remap handling from
__folio_split() into the split helpers. Only anon folios need to
be remapped, so remap_page() is now only called for anon splits and
the anon check in remap_page() is redundant and can be removed.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 58 ++++++++++++++++++++++++++++++--------------------------
1 file changed, 31 insertions(+), 27 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 43093b9a5bca..aa10a13bc255 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3589,9 +3589,6 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags)
{
int i = 0;
- /* If unmap_folio() uses try_to_migrate() on file, remove this check */
- if (!folio_test_anon(folio))
- return;
for (;;) {
remove_migration_ptes(folio, folio, TTU_RMAP_LOCKED | flags);
i += folio_nr_pages(folio);
@@ -3945,19 +3942,23 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
-static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int new_order,
- struct page *split_at, bool do_lru,
- struct list_head *list, enum split_type split_type)
+static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int new_order,
+ struct page *split_at, bool do_lru, bool unmap,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
+ enum ttu_flags ttu_flags = 0;
struct list_lru_one *lru;
struct lruvec *lruvec;
bool dequeue_deferred;
int ret = 0;
+ if (unmap)
+ unmap_folio(folio);
+
local_irq_disable();
/*
@@ -3982,8 +3983,8 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
list_lru_unlock(lru);
rcu_read_unlock();
}
- local_irq_enable();
- return -EAGAIN;
+ ret = -EAGAIN;
+ goto out_no_split;
}
if (dequeue_deferred) {
@@ -4031,15 +4032,21 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
lruvec_unlock(lruvec);
if (ci)
swap_cluster_unlock(ci);
+out_no_split:
local_irq_enable();
+ if (unmap) {
+ if (!ret && !folio_is_device_private(folio))
+ ttu_flags = TTU_USE_SHARED_ZEROPAGE;
+ remap_page(folio, 1 << old_order, ttu_flags);
+ }
return ret;
}
-static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct xa_state *xas,
- struct address_space *mapping, bool do_lru,
- struct list_head *list, enum split_type split_type)
+static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
+ struct page *split_at, struct xa_state *xas,
+ struct address_space *mapping, bool do_lru,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
@@ -4059,6 +4066,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (shmem_mapping(mapping))
end = shmem_fallocend(mapping->host, end);
+ unmap_folio(folio);
+
xas_lock_irq(xas);
/*
@@ -4139,8 +4148,11 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (do_lru)
lruvec_unlock(lruvec);
-
fail:
+ /*
+ * If we want to use try_to_migrate() on file in unmap_folio,
+ * remember to add remap_page() and adapt it.
+ */
xas_unlock_irq(xas);
if (nr_shmem_dropped)
shmem_uncharge(mapping->host, nr_shmem_dropped);
@@ -4180,7 +4192,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
- enum ttu_flags ttu_flags = 0;
int ret;
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
@@ -4268,21 +4279,14 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
goto out_unlock;
}
- unmap_folio(folio);
-
if (!is_anon) {
- ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
- true, list, split_type);
+ ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at, &xas, mapping,
+ true, list, split_type);
} else {
- ret = __folio_freeze_split_unmapped_anon(folio, new_order, split_at, true,
- list, split_type);
+ ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
+ true, list, split_type);
}
- if (!ret && is_anon && !folio_is_device_private(folio))
- ttu_flags = TTU_USE_SHARED_ZEROPAGE;
-
- remap_page(folio, 1 << old_order, ttu_flags);
-
/*
* Drop the mapping while the inode is still pinned. @folio stays
* locked and present in the page cache until the loop below, so
@@ -4365,8 +4369,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
return -EAGAIN;
- return __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
- false, NULL, SPLIT_TYPE_UNIFORM);
+ return __folio_split_unmap_and_freeze_anon(folio, new_order, &folio->page, false,
+ false, NULL, SPLIT_TYPE_UNIFORM);
}
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 08/17] mm/huge_memory: move unmap and remap into the split helpers
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
To prepare for further cleanup, move the unmap/remap handling from
__folio_split() into the split helpers. Only anon folios need to
be remapped, so remap_page() is now only called for anon splits and
the anon check in remap_page() is redundant and can be removed.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 58 ++++++++++++++++++++++++++++++--------------------------
1 file changed, 31 insertions(+), 27 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 43093b9a5bca..aa10a13bc255 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3589,9 +3589,6 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags)
{
int i = 0;
- /* If unmap_folio() uses try_to_migrate() on file, remove this check */
- if (!folio_test_anon(folio))
- return;
for (;;) {
remove_migration_ptes(folio, folio, TTU_RMAP_LOCKED | flags);
i += folio_nr_pages(folio);
@@ -3945,19 +3942,23 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
-static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int new_order,
- struct page *split_at, bool do_lru,
- struct list_head *list, enum split_type split_type)
+static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int new_order,
+ struct page *split_at, bool do_lru, bool unmap,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
+ enum ttu_flags ttu_flags = 0;
struct list_lru_one *lru;
struct lruvec *lruvec;
bool dequeue_deferred;
int ret = 0;
+ if (unmap)
+ unmap_folio(folio);
+
local_irq_disable();
/*
@@ -3982,8 +3983,8 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
list_lru_unlock(lru);
rcu_read_unlock();
}
- local_irq_enable();
- return -EAGAIN;
+ ret = -EAGAIN;
+ goto out_no_split;
}
if (dequeue_deferred) {
@@ -4031,15 +4032,21 @@ static int __folio_freeze_split_unmapped_anon(struct folio *folio, unsigned int
lruvec_unlock(lruvec);
if (ci)
swap_cluster_unlock(ci);
+out_no_split:
local_irq_enable();
+ if (unmap) {
+ if (!ret && !folio_is_device_private(folio))
+ ttu_flags = TTU_USE_SHARED_ZEROPAGE;
+ remap_page(folio, 1 << old_order, ttu_flags);
+ }
return ret;
}
-static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct xa_state *xas,
- struct address_space *mapping, bool do_lru,
- struct list_head *list, enum split_type split_type)
+static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
+ struct page *split_at, struct xa_state *xas,
+ struct address_space *mapping, bool do_lru,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
@@ -4059,6 +4066,8 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (shmem_mapping(mapping))
end = shmem_fallocend(mapping->host, end);
+ unmap_folio(folio);
+
xas_lock_irq(xas);
/*
@@ -4139,8 +4148,11 @@ static int __folio_freeze_split_unmapped_file(struct folio *folio, unsigned int
if (do_lru)
lruvec_unlock(lruvec);
-
fail:
+ /*
+ * If we want to use try_to_migrate() on file in unmap_folio,
+ * remember to add remap_page() and adapt it.
+ */
xas_unlock_irq(xas);
if (nr_shmem_dropped)
shmem_uncharge(mapping->host, nr_shmem_dropped);
@@ -4180,7 +4192,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
- enum ttu_flags ttu_flags = 0;
int ret;
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
@@ -4268,21 +4279,14 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
goto out_unlock;
}
- unmap_folio(folio);
-
if (!is_anon) {
- ret = __folio_freeze_split_unmapped_file(folio, new_order, split_at, &xas, mapping,
- true, list, split_type);
+ ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at, &xas, mapping,
+ true, list, split_type);
} else {
- ret = __folio_freeze_split_unmapped_anon(folio, new_order, split_at, true,
- list, split_type);
+ ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
+ true, list, split_type);
}
- if (!ret && is_anon && !folio_is_device_private(folio))
- ttu_flags = TTU_USE_SHARED_ZEROPAGE;
-
- remap_page(folio, 1 << old_order, ttu_flags);
-
/*
* Drop the mapping while the inode is still pinned. @folio stays
* locked and present in the page cache until the loop below, so
@@ -4365,8 +4369,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
return -EAGAIN;
- return __folio_freeze_split_unmapped_anon(folio, new_order, &folio->page,
- false, NULL, SPLIT_TYPE_UNIFORM);
+ return __folio_split_unmap_and_freeze_anon(folio, new_order, &folio->page, false,
+ false, NULL, SPLIT_TYPE_UNIFORM);
}
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 09/17] mm/huge_memory: move anon_vma and filemap management into split helpers
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Only anon split needs vma info, and only file split needs the filemap
handling. Move the related code into separate helpers so they are
genuinely more self-contained.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 177 +++++++++++++++++++++++++------------------------------
1 file changed, 79 insertions(+), 98 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index aa10a13bc255..1a3ca2606c60 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3950,12 +3950,33 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
+ struct anon_vma *anon_vma = NULL;
enum ttu_flags ttu_flags = 0;
struct list_lru_one *lru;
struct lruvec *lruvec;
bool dequeue_deferred;
int ret = 0;
+ /*
+ * Unmap/remap needs the anon_vma. The caller does not necessarily
+ * hold an mmap_lock that would prevent the anon_vma from
+ * disappearing, so we first take a reference and lock it. This is
+ * similar to folio_lock_anon_vma_read() except the write lock is
+ * taken to serialize against parallel split or collapse.
+ */
+ if (unmap) {
+ anon_vma = folio_get_anon_vma(folio);
+ if (!anon_vma)
+ return -EBUSY;
+ anon_vma_lock_write(anon_vma);
+ }
+
+ /* Racy check if we can split the page, before the optional unmap. */
+ if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
+ ret = -EAGAIN;
+ goto out_unlock;
+ }
+
if (unmap)
unmap_folio(folio);
@@ -4039,21 +4060,58 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
ttu_flags = TTU_USE_SHARED_ZEROPAGE;
remap_page(folio, 1 << old_order, ttu_flags);
}
+out_unlock:
+ if (anon_vma) {
+ anon_vma_unlock_write(anon_vma);
+ put_anon_vma(anon_vma);
+ }
return ret;
}
static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct xa_state *xas,
- struct address_space *mapping, bool do_lru,
+ struct page *split_at, bool do_lru,
struct list_head *list, enum split_type split_type)
{
+ struct address_space *mapping = folio->mapping;
+ XA_STATE(xas, &mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
+ unsigned int min_order;
struct lruvec *lruvec;
pgoff_t end = 0;
- int ret;
+ gfp_t gfp;
+ int ret = 0;
+
+ min_order = mapping_min_folio_order(mapping);
+ if (new_order < min_order)
+ return -EINVAL;
+
+ gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK);
+ if (!filemap_release_folio(folio, gfp))
+ return -EBUSY;
+
+ mapping_set_update(&xas, mapping);
+
+ if (split_type == SPLIT_TYPE_UNIFORM) {
+ int old_order = folio_order(folio);
+
+ xas_set_order(&xas, folio->index, new_order);
+ xas_split_alloc(&xas, folio, old_order, gfp);
+ if (xas_error(&xas)) {
+ ret = xas_error(&xas);
+ goto fail_free;
+ }
+ }
+
+ i_mmap_lock_read(mapping);
+
+ /* Racy check if we can split the page, before unmap_folio() */
+ if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
+ ret = -EAGAIN;
+ goto fail_mmap_unlock;
+ }
/*
* __split_frozen_folio() may need to trim off pages beyond
@@ -4068,13 +4126,13 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
unmap_folio(folio);
- xas_lock_irq(xas);
+ xas_lock_irq(&xas);
/*
* Check if the folio is present in page cache.
* We assume all tail are present too, if folio is there.
*/
- if (xas_load(xas) != folio) {
+ if (xas_load(&xas) != folio) {
ret = -EAGAIN;
goto fail;
}
@@ -4101,7 +4159,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
if (do_lru)
lruvec = folio_lruvec_lock(folio);
- ret = __split_frozen_folio(folio, new_order, split_at, xas,
+ ret = __split_frozen_folio(folio, new_order, split_at, &xas,
mapping, split_type);
/*
@@ -4153,9 +4211,19 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
* If we want to use try_to_migrate() on file in unmap_folio,
* remember to add remap_page() and adapt it.
*/
- xas_unlock_irq(xas);
+ xas_unlock_irq(&xas);
+fail_mmap_unlock:
if (nr_shmem_dropped)
shmem_uncharge(mapping->host, nr_shmem_dropped);
+ /*
+ * Drop the mapping while the inode is still pinned. @folio stays
+ * locked and present in the page cache, so eviction cannot free
+ * the inode yet, nothing past this point may touch the inode or
+ * the mapping.
+ */
+ i_mmap_unlock_read(mapping);
+fail_free:
+ xas_destroy(&xas);
return ret;
}
@@ -4184,12 +4252,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct page *split_at, struct page *lock_at,
struct list_head *list, enum split_type split_type)
{
- XA_STATE(xas, &folio->mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
bool is_anon = folio_test_anon(folio);
struct mem_cgroup *memcg, *old_memcg;
- struct address_space *mapping = NULL;
- struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
int ret;
@@ -4220,84 +4285,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
memcg = get_mem_cgroup_from_folio(folio);
old_memcg = set_active_memcg(memcg);
- if (is_anon) {
- /*
- * The caller does not necessarily hold an mmap_lock that would
- * prevent the anon_vma disappearing so we first we take a
- * reference to it and then lock the anon_vma for write. This
- * is similar to folio_lock_anon_vma_read except the write lock
- * is taken to serialise against parallel split or collapse
- * operations.
- */
- anon_vma = folio_get_anon_vma(folio);
- if (!anon_vma) {
- ret = -EBUSY;
- goto out;
- }
- anon_vma_lock_write(anon_vma);
- mapping = NULL;
- } else {
- unsigned int min_order;
- gfp_t gfp;
-
- mapping = folio->mapping;
- min_order = mapping_min_folio_order(mapping);
- if (new_order < min_order) {
- ret = -EINVAL;
- goto out;
- }
-
- gfp = current_gfp_context(mapping_gfp_mask(mapping) &
- GFP_RECLAIM_MASK);
-
- if (!filemap_release_folio(folio, gfp)) {
- ret = -EBUSY;
- goto out;
- }
-
- mapping_set_update(&xas, mapping);
-
- if (split_type == SPLIT_TYPE_UNIFORM) {
- xas_set_order(&xas, folio->index, new_order);
- xas_split_alloc(&xas, folio, old_order, gfp);
- if (xas_error(&xas)) {
- ret = xas_error(&xas);
- goto out;
- }
- }
-
- anon_vma = NULL;
- i_mmap_lock_read(mapping);
- }
-
- /*
- * Racy check if we can split the page, before unmap_folio() will
- * split PMDs
- */
- if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
- ret = -EAGAIN;
- goto out_unlock;
- }
-
- if (!is_anon) {
- ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at, &xas, mapping,
- true, list, split_type);
- } else {
+ if (is_anon)
ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
true, list, split_type);
- }
-
- /*
- * Drop the mapping while the inode is still pinned. @folio stays
- * locked and present in the page cache until the loop below, so
- * eviction cannot free the inode yet; @lock_at is not enough, it may
- * be a tail beyond EOF that the split already dropped from the page
- * cache. Nothing past this point may touch the inode or the mapping.
- */
- if (mapping) {
- i_mmap_unlock_read(mapping);
- mapping = NULL;
- }
+ else
+ ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at,
+ true, list, split_type);
/*
* Unlock all after-split folios except the one containing
@@ -4318,19 +4311,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
free_folio_and_swap_cache(new_folio);
}
-out_unlock:
- if (anon_vma) {
- anon_vma_unlock_write(anon_vma);
- put_anon_vma(anon_vma);
- }
- if (mapping)
- i_mmap_unlock_read(mapping);
-out:
/* restore to caller's old_memcg */
set_active_memcg(old_memcg);
mem_cgroup_put(memcg);
out_no_memcg:
- xas_destroy(&xas);
if (is_pmd_order(old_order))
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
@@ -4366,9 +4350,6 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
- if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
- return -EAGAIN;
-
return __folio_split_unmap_and_freeze_anon(folio, new_order, &folio->page, false,
false, NULL, SPLIT_TYPE_UNIFORM);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 09/17] mm/huge_memory: move anon_vma and filemap management into split helpers
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Only anon split needs vma info, and only file split needs the filemap
handling. Move the related code into separate helpers so they are
genuinely more self-contained.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 177 +++++++++++++++++++++++++------------------------------
1 file changed, 79 insertions(+), 98 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index aa10a13bc255..1a3ca2606c60 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3950,12 +3950,33 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
+ struct anon_vma *anon_vma = NULL;
enum ttu_flags ttu_flags = 0;
struct list_lru_one *lru;
struct lruvec *lruvec;
bool dequeue_deferred;
int ret = 0;
+ /*
+ * Unmap/remap needs the anon_vma. The caller does not necessarily
+ * hold an mmap_lock that would prevent the anon_vma from
+ * disappearing, so we first take a reference and lock it. This is
+ * similar to folio_lock_anon_vma_read() except the write lock is
+ * taken to serialize against parallel split or collapse.
+ */
+ if (unmap) {
+ anon_vma = folio_get_anon_vma(folio);
+ if (!anon_vma)
+ return -EBUSY;
+ anon_vma_lock_write(anon_vma);
+ }
+
+ /* Racy check if we can split the page, before the optional unmap. */
+ if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
+ ret = -EAGAIN;
+ goto out_unlock;
+ }
+
if (unmap)
unmap_folio(folio);
@@ -4039,21 +4060,58 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
ttu_flags = TTU_USE_SHARED_ZEROPAGE;
remap_page(folio, 1 << old_order, ttu_flags);
}
+out_unlock:
+ if (anon_vma) {
+ anon_vma_unlock_write(anon_vma);
+ put_anon_vma(anon_vma);
+ }
return ret;
}
static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct xa_state *xas,
- struct address_space *mapping, bool do_lru,
+ struct page *split_at, bool do_lru,
struct list_head *list, enum split_type split_type)
{
+ struct address_space *mapping = folio->mapping;
+ XA_STATE(xas, &mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
+ unsigned int min_order;
struct lruvec *lruvec;
pgoff_t end = 0;
- int ret;
+ gfp_t gfp;
+ int ret = 0;
+
+ min_order = mapping_min_folio_order(mapping);
+ if (new_order < min_order)
+ return -EINVAL;
+
+ gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK);
+ if (!filemap_release_folio(folio, gfp))
+ return -EBUSY;
+
+ mapping_set_update(&xas, mapping);
+
+ if (split_type == SPLIT_TYPE_UNIFORM) {
+ int old_order = folio_order(folio);
+
+ xas_set_order(&xas, folio->index, new_order);
+ xas_split_alloc(&xas, folio, old_order, gfp);
+ if (xas_error(&xas)) {
+ ret = xas_error(&xas);
+ goto fail_free;
+ }
+ }
+
+ i_mmap_lock_read(mapping);
+
+ /* Racy check if we can split the page, before unmap_folio() */
+ if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
+ ret = -EAGAIN;
+ goto fail_mmap_unlock;
+ }
/*
* __split_frozen_folio() may need to trim off pages beyond
@@ -4068,13 +4126,13 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
unmap_folio(folio);
- xas_lock_irq(xas);
+ xas_lock_irq(&xas);
/*
* Check if the folio is present in page cache.
* We assume all tail are present too, if folio is there.
*/
- if (xas_load(xas) != folio) {
+ if (xas_load(&xas) != folio) {
ret = -EAGAIN;
goto fail;
}
@@ -4101,7 +4159,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
if (do_lru)
lruvec = folio_lruvec_lock(folio);
- ret = __split_frozen_folio(folio, new_order, split_at, xas,
+ ret = __split_frozen_folio(folio, new_order, split_at, &xas,
mapping, split_type);
/*
@@ -4153,9 +4211,19 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
* If we want to use try_to_migrate() on file in unmap_folio,
* remember to add remap_page() and adapt it.
*/
- xas_unlock_irq(xas);
+ xas_unlock_irq(&xas);
+fail_mmap_unlock:
if (nr_shmem_dropped)
shmem_uncharge(mapping->host, nr_shmem_dropped);
+ /*
+ * Drop the mapping while the inode is still pinned. @folio stays
+ * locked and present in the page cache, so eviction cannot free
+ * the inode yet, nothing past this point may touch the inode or
+ * the mapping.
+ */
+ i_mmap_unlock_read(mapping);
+fail_free:
+ xas_destroy(&xas);
return ret;
}
@@ -4184,12 +4252,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct page *split_at, struct page *lock_at,
struct list_head *list, enum split_type split_type)
{
- XA_STATE(xas, &folio->mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
bool is_anon = folio_test_anon(folio);
struct mem_cgroup *memcg, *old_memcg;
- struct address_space *mapping = NULL;
- struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
int ret;
@@ -4220,84 +4285,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
memcg = get_mem_cgroup_from_folio(folio);
old_memcg = set_active_memcg(memcg);
- if (is_anon) {
- /*
- * The caller does not necessarily hold an mmap_lock that would
- * prevent the anon_vma disappearing so we first we take a
- * reference to it and then lock the anon_vma for write. This
- * is similar to folio_lock_anon_vma_read except the write lock
- * is taken to serialise against parallel split or collapse
- * operations.
- */
- anon_vma = folio_get_anon_vma(folio);
- if (!anon_vma) {
- ret = -EBUSY;
- goto out;
- }
- anon_vma_lock_write(anon_vma);
- mapping = NULL;
- } else {
- unsigned int min_order;
- gfp_t gfp;
-
- mapping = folio->mapping;
- min_order = mapping_min_folio_order(mapping);
- if (new_order < min_order) {
- ret = -EINVAL;
- goto out;
- }
-
- gfp = current_gfp_context(mapping_gfp_mask(mapping) &
- GFP_RECLAIM_MASK);
-
- if (!filemap_release_folio(folio, gfp)) {
- ret = -EBUSY;
- goto out;
- }
-
- mapping_set_update(&xas, mapping);
-
- if (split_type == SPLIT_TYPE_UNIFORM) {
- xas_set_order(&xas, folio->index, new_order);
- xas_split_alloc(&xas, folio, old_order, gfp);
- if (xas_error(&xas)) {
- ret = xas_error(&xas);
- goto out;
- }
- }
-
- anon_vma = NULL;
- i_mmap_lock_read(mapping);
- }
-
- /*
- * Racy check if we can split the page, before unmap_folio() will
- * split PMDs
- */
- if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
- ret = -EAGAIN;
- goto out_unlock;
- }
-
- if (!is_anon) {
- ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at, &xas, mapping,
- true, list, split_type);
- } else {
+ if (is_anon)
ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
true, list, split_type);
- }
-
- /*
- * Drop the mapping while the inode is still pinned. @folio stays
- * locked and present in the page cache until the loop below, so
- * eviction cannot free the inode yet; @lock_at is not enough, it may
- * be a tail beyond EOF that the split already dropped from the page
- * cache. Nothing past this point may touch the inode or the mapping.
- */
- if (mapping) {
- i_mmap_unlock_read(mapping);
- mapping = NULL;
- }
+ else
+ ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at,
+ true, list, split_type);
/*
* Unlock all after-split folios except the one containing
@@ -4318,19 +4311,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
free_folio_and_swap_cache(new_folio);
}
-out_unlock:
- if (anon_vma) {
- anon_vma_unlock_write(anon_vma);
- put_anon_vma(anon_vma);
- }
- if (mapping)
- i_mmap_unlock_read(mapping);
-out:
/* restore to caller's old_memcg */
set_active_memcg(old_memcg);
mem_cgroup_put(memcg);
out_no_memcg:
- xas_destroy(&xas);
if (is_pmd_order(old_order))
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
@@ -4366,9 +4350,6 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
- if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
- return -EAGAIN;
-
return __folio_split_unmap_and_freeze_anon(folio, new_order, &folio->page, false,
false, NULL, SPLIT_TYPE_UNIFORM);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 10/17] mm/huge_memory: move memcg switch into the file split helper
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
The xarray node allocations in __folio_freeze_split_unmap_file() need
to be charged to the folio's memcg, so move the memcg switch from
__folio_split() into the helper.
The anon split helper and the after-split folio freeing perform no
chargeable allocations, so no memcg handling is left in __folio_split().
Rename its out_no_memcg label to out.
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 36 +++++++++++++++++++-----------------
1 file changed, 19 insertions(+), 17 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1a3ca2606c60..29b46f039d1e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4076,6 +4076,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
struct address_space *mapping = folio->mapping;
XA_STATE(xas, &mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
+ struct mem_cgroup *memcg, *old_memcg;
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
unsigned int min_order;
@@ -4088,9 +4089,18 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
if (new_order < min_order)
return -EINVAL;
+ /*
+ * Switch to folio's memcg as xarray node allocation can happen and
+ * needs to charge to it.
+ */
+ memcg = get_mem_cgroup_from_folio(folio);
+ old_memcg = set_active_memcg(memcg);
+
gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK);
- if (!filemap_release_folio(folio, gfp))
- return -EBUSY;
+ if (!filemap_release_folio(folio, gfp)) {
+ ret = -EBUSY;
+ goto fail_free;
+ }
mapping_set_update(&xas, mapping);
@@ -4223,6 +4233,9 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
*/
i_mmap_unlock_read(mapping);
fail_free:
+ /* Restore the previously active memcg */
+ set_active_memcg(old_memcg);
+ mem_cgroup_put(memcg);
xas_destroy(&xas);
return ret;
}
@@ -4254,7 +4267,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
{
struct folio *end_folio = folio_next(folio);
bool is_anon = folio_test_anon(folio);
- struct mem_cgroup *memcg, *old_memcg;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
int ret;
@@ -4264,27 +4276,20 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
ret = -EINVAL;
- goto out_no_memcg;
+ goto out;
}
if (new_order >= old_order) {
ret = -EINVAL;
- goto out_no_memcg;
+ goto out;
}
ret = folio_check_splittable(folio, new_order, split_type);
if (ret) {
VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
- goto out_no_memcg;
+ goto out;
}
- /*
- * switch to folio's memcg as xarray node allocation can happen and
- * needs to charge to it.
- */
- memcg = get_mem_cgroup_from_folio(folio);
- old_memcg = set_active_memcg(memcg);
-
if (is_anon)
ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
true, list, split_type);
@@ -4311,10 +4316,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
free_folio_and_swap_cache(new_folio);
}
- /* restore to caller's old_memcg */
- set_active_memcg(old_memcg);
- mem_cgroup_put(memcg);
-out_no_memcg:
+out:
if (is_pmd_order(old_order))
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 10/17] mm/huge_memory: move memcg switch into the file split helper
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
The xarray node allocations in __folio_freeze_split_unmap_file() need
to be charged to the folio's memcg, so move the memcg switch from
__folio_split() into the helper.
The anon split helper and the after-split folio freeing perform no
chargeable allocations, so no memcg handling is left in __folio_split().
Rename its out_no_memcg label to out.
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 36 +++++++++++++++++++-----------------
1 file changed, 19 insertions(+), 17 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1a3ca2606c60..29b46f039d1e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4076,6 +4076,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
struct address_space *mapping = folio->mapping;
XA_STATE(xas, &mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
+ struct mem_cgroup *memcg, *old_memcg;
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
unsigned int min_order;
@@ -4088,9 +4089,18 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
if (new_order < min_order)
return -EINVAL;
+ /*
+ * Switch to folio's memcg as xarray node allocation can happen and
+ * needs to charge to it.
+ */
+ memcg = get_mem_cgroup_from_folio(folio);
+ old_memcg = set_active_memcg(memcg);
+
gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK);
- if (!filemap_release_folio(folio, gfp))
- return -EBUSY;
+ if (!filemap_release_folio(folio, gfp)) {
+ ret = -EBUSY;
+ goto fail_free;
+ }
mapping_set_update(&xas, mapping);
@@ -4223,6 +4233,9 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
*/
i_mmap_unlock_read(mapping);
fail_free:
+ /* Restore the previously active memcg */
+ set_active_memcg(old_memcg);
+ mem_cgroup_put(memcg);
xas_destroy(&xas);
return ret;
}
@@ -4254,7 +4267,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
{
struct folio *end_folio = folio_next(folio);
bool is_anon = folio_test_anon(folio);
- struct mem_cgroup *memcg, *old_memcg;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
int ret;
@@ -4264,27 +4276,20 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
ret = -EINVAL;
- goto out_no_memcg;
+ goto out;
}
if (new_order >= old_order) {
ret = -EINVAL;
- goto out_no_memcg;
+ goto out;
}
ret = folio_check_splittable(folio, new_order, split_type);
if (ret) {
VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
- goto out_no_memcg;
+ goto out;
}
- /*
- * switch to folio's memcg as xarray node allocation can happen and
- * needs to charge to it.
- */
- memcg = get_mem_cgroup_from_folio(folio);
- old_memcg = set_active_memcg(memcg);
-
if (is_anon)
ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
true, list, split_type);
@@ -4311,10 +4316,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
free_folio_and_swap_cache(new_folio);
}
- /* restore to caller's old_memcg */
- set_active_memcg(old_memcg);
- mem_cgroup_put(memcg);
-out_no_memcg:
+out:
if (is_pmd_order(old_order))
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 11/17] mm/huge_memory: allow splitting mappingless swap cache folios
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Lift the restriction that kept swap cache folios without a mapping
from being split. All the underlying infrastructure is sound against
that with a few more tweaks, no reason to block it anymore.
Also rename the split helper, which now handles mappingless swap
cache folios that are yet to be anon, or may actually belong to
shmem. In either case there is not much difference in how they would
be split.
A non-anon swap cache folio that still has a mapping (e.g. a shmem
swap cache folio) remains rejected up front: it would need both its
page cache and swap cache entries updated on split, which the split
helpers do not do, and there would be little benefit in doing so.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 40 +++++++++++++++++++++++-----------------
1 file changed, 23 insertions(+), 17 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 29b46f039d1e..67401c58794e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3893,12 +3893,10 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
/*
* Folios that just got truncated cannot get split. Signal to the
- * caller that there was a race.
- *
- * TODO: this will also currently refuse folios without a mapping in the
- * swapcache (shmem or to-be-anon folios).
+ * caller that there was a race. A mappingless swap cache folio
+ * has no page cache entries to update, so it is fine to split.
*/
- if (!folio->mapping && !is_anon)
+ if (!folio->mapping && !is_swapcache)
return -EBUSY;
/* order-1 is not supported for anonymous THP. */
@@ -3942,11 +3940,12 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
-static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int new_order,
- struct page *split_at, bool do_lru, bool unmap,
- struct list_head *list, enum split_type split_type)
+static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_order,
+ struct page *split_at, bool do_lru, bool anon_unmap,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
+ bool is_anon = folio_test_anon(folio);
struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
@@ -3964,7 +3963,7 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
* similar to folio_lock_anon_vma_read() except the write lock is
* taken to serialize against parallel split or collapse.
*/
- if (unmap) {
+ if (anon_unmap) {
anon_vma = folio_get_anon_vma(folio);
if (!anon_vma)
return -EBUSY;
@@ -3977,7 +3976,7 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
goto out_unlock;
}
- if (unmap)
+ if (anon_unmap)
unmap_folio(folio);
local_irq_disable();
@@ -3988,8 +3987,11 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
* a 0-ref folio, it assumes it beat folio_put() to the list
* lock and must clean up the LRU state - the same dequeue we
* will do below as part of the split.
+ *
+ * Only anon folios are ever queued on the deferred split list,
+ * so non-anon folios (mappingless swapcache) never need dequeuing.
*/
- dequeue_deferred = old_order > 1;
+ dequeue_deferred = old_order > 1 && is_anon;
if (dequeue_deferred) {
struct mem_cgroup *memcg;
@@ -4055,13 +4057,13 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
swap_cluster_unlock(ci);
out_no_split:
local_irq_enable();
- if (unmap) {
+ if (anon_unmap) {
if (!ret && !folio_is_device_private(folio))
ttu_flags = TTU_USE_SHARED_ZEROPAGE;
remap_page(folio, 1 << old_order, ttu_flags);
}
out_unlock:
- if (anon_vma) {
+ if (anon_unmap) {
anon_vma_unlock_write(anon_vma);
put_anon_vma(anon_vma);
}
@@ -4265,6 +4267,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct page *split_at, struct page *lock_at,
struct list_head *list, enum split_type split_type)
{
+ bool is_swapcache = folio_test_swapcache(folio);
struct folio *end_folio = folio_next(folio);
bool is_anon = folio_test_anon(folio);
int old_order = folio_order(folio);
@@ -4291,8 +4294,11 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
}
if (is_anon)
- ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
- true, list, split_type);
+ ret = __folio_split_unmap_and_freeze(folio, new_order, split_at, true,
+ true, list, split_type);
+ else if (is_swapcache)
+ ret = __folio_split_unmap_and_freeze(folio, new_order, split_at, true,
+ false, list, split_type);
else
ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at,
true, list, split_type);
@@ -4352,8 +4358,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
- return __folio_split_unmap_and_freeze_anon(folio, new_order, &folio->page, false,
- false, NULL, SPLIT_TYPE_UNIFORM);
+ return __folio_split_unmap_and_freeze(folio, new_order, &folio->page, false,
+ false, NULL, SPLIT_TYPE_UNIFORM);
}
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 11/17] mm/huge_memory: allow splitting mappingless swap cache folios
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Lift the restriction that kept swap cache folios without a mapping
from being split. All the underlying infrastructure is sound against
that with a few more tweaks, no reason to block it anymore.
Also rename the split helper, which now handles mappingless swap
cache folios that are yet to be anon, or may actually belong to
shmem. In either case there is not much difference in how they would
be split.
A non-anon swap cache folio that still has a mapping (e.g. a shmem
swap cache folio) remains rejected up front: it would need both its
page cache and swap cache entries updated on split, which the split
helpers do not do, and there would be little benefit in doing so.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 40 +++++++++++++++++++++++-----------------
1 file changed, 23 insertions(+), 17 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 29b46f039d1e..67401c58794e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3893,12 +3893,10 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
/*
* Folios that just got truncated cannot get split. Signal to the
- * caller that there was a race.
- *
- * TODO: this will also currently refuse folios without a mapping in the
- * swapcache (shmem or to-be-anon folios).
+ * caller that there was a race. A mappingless swap cache folio
+ * has no page cache entries to update, so it is fine to split.
*/
- if (!folio->mapping && !is_anon)
+ if (!folio->mapping && !is_swapcache)
return -EBUSY;
/* order-1 is not supported for anonymous THP. */
@@ -3942,11 +3940,12 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
-static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int new_order,
- struct page *split_at, bool do_lru, bool unmap,
- struct list_head *list, enum split_type split_type)
+static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_order,
+ struct page *split_at, bool do_lru, bool anon_unmap,
+ struct list_head *list, enum split_type split_type)
{
struct folio *end_folio = folio_next(folio);
+ bool is_anon = folio_test_anon(folio);
struct swap_cluster_info *ci = NULL;
struct folio *new_folio, *next;
int old_order = folio_order(folio);
@@ -3964,7 +3963,7 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
* similar to folio_lock_anon_vma_read() except the write lock is
* taken to serialize against parallel split or collapse.
*/
- if (unmap) {
+ if (anon_unmap) {
anon_vma = folio_get_anon_vma(folio);
if (!anon_vma)
return -EBUSY;
@@ -3977,7 +3976,7 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
goto out_unlock;
}
- if (unmap)
+ if (anon_unmap)
unmap_folio(folio);
local_irq_disable();
@@ -3988,8 +3987,11 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
* a 0-ref folio, it assumes it beat folio_put() to the list
* lock and must clean up the LRU state - the same dequeue we
* will do below as part of the split.
+ *
+ * Only anon folios are ever queued on the deferred split list,
+ * so non-anon folios (mappingless swapcache) never need dequeuing.
*/
- dequeue_deferred = old_order > 1;
+ dequeue_deferred = old_order > 1 && is_anon;
if (dequeue_deferred) {
struct mem_cgroup *memcg;
@@ -4055,13 +4057,13 @@ static int __folio_split_unmap_and_freeze_anon(struct folio *folio, unsigned int
swap_cluster_unlock(ci);
out_no_split:
local_irq_enable();
- if (unmap) {
+ if (anon_unmap) {
if (!ret && !folio_is_device_private(folio))
ttu_flags = TTU_USE_SHARED_ZEROPAGE;
remap_page(folio, 1 << old_order, ttu_flags);
}
out_unlock:
- if (anon_vma) {
+ if (anon_unmap) {
anon_vma_unlock_write(anon_vma);
put_anon_vma(anon_vma);
}
@@ -4265,6 +4267,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct page *split_at, struct page *lock_at,
struct list_head *list, enum split_type split_type)
{
+ bool is_swapcache = folio_test_swapcache(folio);
struct folio *end_folio = folio_next(folio);
bool is_anon = folio_test_anon(folio);
int old_order = folio_order(folio);
@@ -4291,8 +4294,11 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
}
if (is_anon)
- ret = __folio_split_unmap_and_freeze_anon(folio, new_order, split_at, true,
- true, list, split_type);
+ ret = __folio_split_unmap_and_freeze(folio, new_order, split_at, true,
+ true, list, split_type);
+ else if (is_swapcache)
+ ret = __folio_split_unmap_and_freeze(folio, new_order, split_at, true,
+ false, list, split_type);
else
ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at,
true, list, split_type);
@@ -4352,8 +4358,8 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
- return __folio_split_unmap_and_freeze_anon(folio, new_order, &folio->page, false,
- false, NULL, SPLIT_TYPE_UNIFORM);
+ return __folio_split_unmap_and_freeze(folio, new_order, &folio->page, false,
+ false, NULL, SPLIT_TYPE_UNIFORM);
}
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 12/17] mm/huge_memory: add kerneldoc for the split helpers
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Document __folio_split_unmap_and_freeze() and
__folio_split_unmap_and_freeze_file(), and rename the file split
helper's definition to match its call site.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 67401c58794e..ce02608b37f4 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3940,6 +3940,25 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
+/**
+ * __folio_split_unmap_and_freeze() - split an anon or swap cache folio
+ * @folio: folio to split, must be locked
+ * @new_order: the order of the after-split folios (uniform split), or the
+ * smallest order of the after-split folios (non-uniform split)
+ * @split_at: in non-uniform split, the folio containing @split_at is split
+ * until its order becomes @new_order
+ * @do_lru: if true, add after-split folios to @list if non NULL, otherwise to
+ * the LRU list
+ * @anon_unmap: if true, unmap @folio before the split and remap it after
+ * @list: after-split folios will be put on it if non NULL
+ * @split_type: perform uniform split or not (non-uniform split)
+ *
+ * Helper for splitting an anon or swap cache folio. It unmaps @folio (unless
+ * @anon_unmap is false), freezes its refcount, and performs the split, updates
+ * the swap cache entries. Split folios are unfrozen and remapped.
+ *
+ * Return: 0 on success, otherwise an error number is returned.
+ */
static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_order,
struct page *split_at, bool do_lru, bool anon_unmap,
struct list_head *list, enum split_type split_type)
@@ -4071,6 +4090,25 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
return ret;
}
+/**
+ * __folio_split_unmap_and_freeze_file() - split a file-backed folio
+ * @folio: folio to split, must be locked and file-backed
+ * @new_order: the order of the after-split folios (uniform split), or the
+ * smallest order of the after-split folios (non-uniform split)
+ * @split_at: in non-uniform split, the folio containing @split_at is split
+ * until its order becomes @new_order
+ * @do_lru: if true, add after-split folios to @list if non NULL, otherwise to
+ * the LRU list
+ * @list: after-split folios will be put on it if non NULL
+ * @split_type: perform uniform split or not (non-uniform split)
+ *
+ * Helper for splitting a file-backed folio. It unmaps @folio, freezes its
+ * refcount, and perform the split, updates the page cache entries. Split
+ * folios are unfrozen but not remapped, they are faulted back in on demand.
+ *
+ * Return: 0 on success, otherwise an error number is returned. (if -ENOMEM
+ * is returned, @folio might be split but not to @new_order)
+ */
static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
struct page *split_at, bool do_lru,
struct list_head *list, enum split_type split_type)
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 12/17] mm/huge_memory: add kerneldoc for the split helpers
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Document __folio_split_unmap_and_freeze() and
__folio_split_unmap_and_freeze_file(), and rename the file split
helper's definition to match its call site.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 67401c58794e..ce02608b37f4 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3940,6 +3940,25 @@ static unsigned int folio_cache_ref_count(const struct folio *folio)
return folio_nr_pages(folio);
}
+/**
+ * __folio_split_unmap_and_freeze() - split an anon or swap cache folio
+ * @folio: folio to split, must be locked
+ * @new_order: the order of the after-split folios (uniform split), or the
+ * smallest order of the after-split folios (non-uniform split)
+ * @split_at: in non-uniform split, the folio containing @split_at is split
+ * until its order becomes @new_order
+ * @do_lru: if true, add after-split folios to @list if non NULL, otherwise to
+ * the LRU list
+ * @anon_unmap: if true, unmap @folio before the split and remap it after
+ * @list: after-split folios will be put on it if non NULL
+ * @split_type: perform uniform split or not (non-uniform split)
+ *
+ * Helper for splitting an anon or swap cache folio. It unmaps @folio (unless
+ * @anon_unmap is false), freezes its refcount, and performs the split, updates
+ * the swap cache entries. Split folios are unfrozen and remapped.
+ *
+ * Return: 0 on success, otherwise an error number is returned.
+ */
static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_order,
struct page *split_at, bool do_lru, bool anon_unmap,
struct list_head *list, enum split_type split_type)
@@ -4071,6 +4090,25 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
return ret;
}
+/**
+ * __folio_split_unmap_and_freeze_file() - split a file-backed folio
+ * @folio: folio to split, must be locked and file-backed
+ * @new_order: the order of the after-split folios (uniform split), or the
+ * smallest order of the after-split folios (non-uniform split)
+ * @split_at: in non-uniform split, the folio containing @split_at is split
+ * until its order becomes @new_order
+ * @do_lru: if true, add after-split folios to @list if non NULL, otherwise to
+ * the LRU list
+ * @list: after-split folios will be put on it if non NULL
+ * @split_type: perform uniform split or not (non-uniform split)
+ *
+ * Helper for splitting a file-backed folio. It unmaps @folio, freezes its
+ * refcount, and perform the split, updates the page cache entries. Split
+ * folios are unfrozen but not remapped, they are faulted back in on demand.
+ *
+ * Return: 0 on success, otherwise an error number is returned. (if -ENOMEM
+ * is returned, @folio might be split but not to @new_order)
+ */
static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
struct page *split_at, bool do_lru,
struct list_head *list, enum split_type split_type)
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 13/17] mm/huge_memory: drop the unused do_lru argument of the file split helper
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
The only caller of __folio_split_unmap_and_freeze_file() always passes
do_lru as true, so the argument and the branches gated on it are dead
code. Drop it.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 19 ++++++-------------
1 file changed, 6 insertions(+), 13 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ce02608b37f4..72e7d24139e6 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4097,8 +4097,6 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
* smallest order of the after-split folios (non-uniform split)
* @split_at: in non-uniform split, the folio containing @split_at is split
* until its order becomes @new_order
- * @do_lru: if true, add after-split folios to @list if non NULL, otherwise to
- * the LRU list
* @list: after-split folios will be put on it if non NULL
* @split_type: perform uniform split or not (non-uniform split)
*
@@ -4110,8 +4108,8 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
* is returned, @folio might be split but not to @new_order)
*/
static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
- struct page *split_at, bool do_lru,
- struct list_head *list, enum split_type split_type)
+ struct page *split_at, struct list_head *list,
+ enum split_type split_type)
{
struct address_space *mapping = folio->mapping;
XA_STATE(xas, &mapping->i_pages, folio->index);
@@ -4206,9 +4204,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
}
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
- if (do_lru)
- lruvec = folio_lruvec_lock(folio);
-
+ lruvec = folio_lruvec_lock(folio);
ret = __split_frozen_folio(folio, new_order, split_at, &xas,
mapping, split_type);
@@ -4226,8 +4222,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
folio_ref_unfreeze(new_folio,
folio_cache_ref_count(new_folio) + 1);
- if (do_lru)
- lru_add_split_folio(folio, new_folio, lruvec, list);
+ lru_add_split_folio(folio, new_folio, lruvec, list);
/* Add the new folio to the page cache. */
if (new_folio->index < end) {
@@ -4253,9 +4248,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
* and its caller can see stale page cache entries.
*/
folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
-
- if (do_lru)
- lruvec_unlock(lruvec);
+ lruvec_unlock(lruvec);
fail:
/*
* If we want to use try_to_migrate() on file in unmap_folio,
@@ -4339,7 +4332,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
false, list, split_type);
else
ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at,
- true, list, split_type);
+ list, split_type);
/*
* Unlock all after-split folios except the one containing
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 13/17] mm/huge_memory: drop the unused do_lru argument of the file split helper
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
The only caller of __folio_split_unmap_and_freeze_file() always passes
do_lru as true, so the argument and the branches gated on it are dead
code. Drop it.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 19 ++++++-------------
1 file changed, 6 insertions(+), 13 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ce02608b37f4..72e7d24139e6 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4097,8 +4097,6 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
* smallest order of the after-split folios (non-uniform split)
* @split_at: in non-uniform split, the folio containing @split_at is split
* until its order becomes @new_order
- * @do_lru: if true, add after-split folios to @list if non NULL, otherwise to
- * the LRU list
* @list: after-split folios will be put on it if non NULL
* @split_type: perform uniform split or not (non-uniform split)
*
@@ -4110,8 +4108,8 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
* is returned, @folio might be split but not to @new_order)
*/
static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int new_order,
- struct page *split_at, bool do_lru,
- struct list_head *list, enum split_type split_type)
+ struct page *split_at, struct list_head *list,
+ enum split_type split_type)
{
struct address_space *mapping = folio->mapping;
XA_STATE(xas, &mapping->i_pages, folio->index);
@@ -4206,9 +4204,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
}
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
- if (do_lru)
- lruvec = folio_lruvec_lock(folio);
-
+ lruvec = folio_lruvec_lock(folio);
ret = __split_frozen_folio(folio, new_order, split_at, &xas,
mapping, split_type);
@@ -4226,8 +4222,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
folio_ref_unfreeze(new_folio,
folio_cache_ref_count(new_folio) + 1);
- if (do_lru)
- lru_add_split_folio(folio, new_folio, lruvec, list);
+ lru_add_split_folio(folio, new_folio, lruvec, list);
/* Add the new folio to the page cache. */
if (new_folio->index < end) {
@@ -4253,9 +4248,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
* and its caller can see stale page cache entries.
*/
folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
-
- if (do_lru)
- lruvec_unlock(lruvec);
+ lruvec_unlock(lruvec);
fail:
/*
* If we want to use try_to_migrate() on file in unmap_folio,
@@ -4339,7 +4332,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
false, list, split_type);
else
ret = __folio_split_unmap_and_freeze_file(folio, new_order, split_at,
- true, list, split_type);
+ list, split_type);
/*
* Unlock all after-split folios except the one containing
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 14/17] mm/huge_memory: clean up after-split folio freeing in __folio_split
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Replace free_folio_and_swap_cache() with an explicit folio_free_swap()
and folio_put() in the after-split loop. free_folio_and_swap_cache()
unlocks the folio, then free_swap_cache() must trylock it again and
re-check folio_mapped() before freeing the swap cache entries; if the
trylock loses a race, the entries are left behind even though the folio
reference is dropped. The sub folios are still locked and unmapped
here, so just directly call folio_free_swap() directly under the lock,
unlock and drop the reference. This makes the swap cache freeing
deterministic and the reference drop explicit.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 72e7d24139e6..503e3bd84cad 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4343,14 +4343,16 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (new_folio == page_folio(lock_at))
continue;
- folio_unlock(new_folio);
/*
* Subpages whose mapping has been zapped may be freed
* earlier, but freeing them requires taking the
- * lru_lock, so we defer put_page() on tail pages until
+ * lru_lock, so we defer folio_put() on tail pages until
* after the split completes.
*/
- free_folio_and_swap_cache(new_folio);
+ if (is_swapcache)
+ folio_free_swap(new_folio);
+ folio_unlock(new_folio);
+ folio_put(new_folio);
}
out:
@@ -4377,7 +4379,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
* isolated from LRU (if applicable)
*
* Upon return, the folio is not remapped, split folios are not added to LRU,
- * free_folio_and_swap_cache() is not called, and new folios remain locked.
+ * folio_free_swap() is not called, and new folios remain locked.
*
* Return: 0 on success, -EAGAIN if the folio cannot be split (e.g., due to
* insufficient reference count or extra pins).
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 14/17] mm/huge_memory: clean up after-split folio freeing in __folio_split
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Replace free_folio_and_swap_cache() with an explicit folio_free_swap()
and folio_put() in the after-split loop. free_folio_and_swap_cache()
unlocks the folio, then free_swap_cache() must trylock it again and
re-check folio_mapped() before freeing the swap cache entries; if the
trylock loses a race, the entries are left behind even though the folio
reference is dropped. The sub folios are still locked and unmapped
here, so just directly call folio_free_swap() directly under the lock,
unlock and drop the reference. This makes the swap cache freeing
deterministic and the reference drop explicit.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 72e7d24139e6..503e3bd84cad 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4343,14 +4343,16 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (new_folio == page_folio(lock_at))
continue;
- folio_unlock(new_folio);
/*
* Subpages whose mapping has been zapped may be freed
* earlier, but freeing them requires taking the
- * lru_lock, so we defer put_page() on tail pages until
+ * lru_lock, so we defer folio_put() on tail pages until
* after the split completes.
*/
- free_folio_and_swap_cache(new_folio);
+ if (is_swapcache)
+ folio_free_swap(new_folio);
+ folio_unlock(new_folio);
+ folio_put(new_folio);
}
out:
@@ -4377,7 +4379,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
* isolated from LRU (if applicable)
*
* Upon return, the folio is not remapped, split folios are not added to LRU,
- * free_folio_and_swap_cache() is not called, and new folios remain locked.
+ * folio_free_swap() is not called, and new folios remain locked.
*
* Return: 0 on success, -EAGAIN if the folio cannot be split (e.g., due to
* insufficient reference count or extra pins).
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 15/17] mm/huge_memory: lift order-0 restriction for swapcache split
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
The restriction that swapcache folios can only be uniformly split to
order 0 dates back to when the swap cache was managed via address_space
mapping (swap_address_space). The old split loop only created order-0
sub-folios with a fixed stride, so non-uniform split and non-zero order
were rightfully blocked.
After the swap cache switched to swap table under a cluster lock,
__swap_cache_replace_folio already gained the ability to replace any
number of entries for any sub-folio size in one cluster, and the old
swap_address_space locking and limit was removed. The restriction
became obsolete but persisted through multiple refactorings.
Drop it now: swapcache folios can be split to any supported order with
either uniform or non-uniform split, except order-1 which is not
supported for anon folios. Mappingless swap cache folios could be either
anon or shmem, so for now we just simply forbid order-1 for all swapcache.
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 31 +++++++++++++------------------
1 file changed, 13 insertions(+), 18 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 503e3bd84cad..317e5b63d44b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3809,6 +3809,7 @@ static int __split_frozen_folio(struct folio *folio, int new_order,
struct address_space *mapping, enum split_type split_type)
{
const bool is_anon = folio_test_anon(folio);
+ const bool is_swapcache = folio_test_swapcache(folio);
int old_order = folio_order(folio);
int start_order = split_type == SPLIT_TYPE_UNIFORM ? new_order : old_order - 1;
struct folio *old_folio = folio;
@@ -3823,8 +3824,8 @@ static int __split_frozen_folio(struct folio *folio, int new_order,
split_order--) {
int nr_new_folios = 1UL << (old_order - split_order);
- /* order-1 anonymous folio is not supported */
- if (is_anon && split_order == 1)
+ /* order-1 anonymous or swapcache folio is not supported */
+ if ((is_anon || is_swapcache) && split_order == 1)
continue;
if (mapping) {
@@ -3899,19 +3900,13 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
if (!folio->mapping && !is_swapcache)
return -EBUSY;
- /* order-1 is not supported for anonymous THP. */
- if (is_anon && new_order == 1)
- return -EINVAL;
-
/*
- * swapcache folio could only be split to order 0
- *
- * non-uniform split creates after-split folios with orders from
- * folio_order(folio) - 1 to new_order, making it not suitable for any
- * swapcache folio split. Only uniform split to order-0 can be used
- * here.
+ * Order-1 is unsupported: anon folios need subpage 2 for the
+ * deferred split list, hybrid shmem & swap cache folios are not
+ * splittable, and a splittable mappingless swap cache folio could
+ * be either anon or shmem, which we cannot tell apart.
*/
- if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && is_swapcache)
+ if ((is_anon || is_swapcache) && new_order == 1)
return -EINVAL;
if (is_huge_zero_folio(folio))
@@ -4411,11 +4406,11 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
* GUP pins, will result in the folio not getting split; instead, the caller
* will receive an -EAGAIN.
*
- * 4) @new_order > 1, usually. Splitting to order-1 anonymous folios is not
- * supported for non-file-backed folios, because folio->_deferred_list, which
- * is used by partially mapped folios, is stored in subpage 2, but an order-1
- * folio only has subpages 0 and 1. File-backed order-1 folios are supported,
- * since they do not use _deferred_list.
+ * 4) @new_order > 1, usually. Order-1 is not supported for anon or swapcache
+ * folios: anon folios need subpage 2 for _deferred_list, which order-1
+ * folios lack, and a swapcache folio may become anon once faulted in.
+ * File-backed order-1 folios are supported, since they do not use
+ * _deferred_list.
*
* After splitting, the caller's folio reference will be transferred to @page,
* resulting in a raised refcount of @page after this call. The other pages may
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 15/17] mm/huge_memory: lift order-0 restriction for swapcache split
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
The restriction that swapcache folios can only be uniformly split to
order 0 dates back to when the swap cache was managed via address_space
mapping (swap_address_space). The old split loop only created order-0
sub-folios with a fixed stride, so non-uniform split and non-zero order
were rightfully blocked.
After the swap cache switched to swap table under a cluster lock,
__swap_cache_replace_folio already gained the ability to replace any
number of entries for any sub-folio size in one cluster, and the old
swap_address_space locking and limit was removed. The restriction
became obsolete but persisted through multiple refactorings.
Drop it now: swapcache folios can be split to any supported order with
either uniform or non-uniform split, except order-1 which is not
supported for anon folios. Mappingless swap cache folios could be either
anon or shmem, so for now we just simply forbid order-1 for all swapcache.
Acked-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 31 +++++++++++++------------------
1 file changed, 13 insertions(+), 18 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 503e3bd84cad..317e5b63d44b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3809,6 +3809,7 @@ static int __split_frozen_folio(struct folio *folio, int new_order,
struct address_space *mapping, enum split_type split_type)
{
const bool is_anon = folio_test_anon(folio);
+ const bool is_swapcache = folio_test_swapcache(folio);
int old_order = folio_order(folio);
int start_order = split_type == SPLIT_TYPE_UNIFORM ? new_order : old_order - 1;
struct folio *old_folio = folio;
@@ -3823,8 +3824,8 @@ static int __split_frozen_folio(struct folio *folio, int new_order,
split_order--) {
int nr_new_folios = 1UL << (old_order - split_order);
- /* order-1 anonymous folio is not supported */
- if (is_anon && split_order == 1)
+ /* order-1 anonymous or swapcache folio is not supported */
+ if ((is_anon || is_swapcache) && split_order == 1)
continue;
if (mapping) {
@@ -3899,19 +3900,13 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
if (!folio->mapping && !is_swapcache)
return -EBUSY;
- /* order-1 is not supported for anonymous THP. */
- if (is_anon && new_order == 1)
- return -EINVAL;
-
/*
- * swapcache folio could only be split to order 0
- *
- * non-uniform split creates after-split folios with orders from
- * folio_order(folio) - 1 to new_order, making it not suitable for any
- * swapcache folio split. Only uniform split to order-0 can be used
- * here.
+ * Order-1 is unsupported: anon folios need subpage 2 for the
+ * deferred split list, hybrid shmem & swap cache folios are not
+ * splittable, and a splittable mappingless swap cache folio could
+ * be either anon or shmem, which we cannot tell apart.
*/
- if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && is_swapcache)
+ if ((is_anon || is_swapcache) && new_order == 1)
return -EINVAL;
if (is_huge_zero_folio(folio))
@@ -4411,11 +4406,11 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
* GUP pins, will result in the folio not getting split; instead, the caller
* will receive an -EAGAIN.
*
- * 4) @new_order > 1, usually. Splitting to order-1 anonymous folios is not
- * supported for non-file-backed folios, because folio->_deferred_list, which
- * is used by partially mapped folios, is stored in subpage 2, but an order-1
- * folio only has subpages 0 and 1. File-backed order-1 folios are supported,
- * since they do not use _deferred_list.
+ * 4) @new_order > 1, usually. Order-1 is not supported for anon or swapcache
+ * folios: anon folios need subpage 2 for _deferred_list, which order-1
+ * folios lack, and a swapcache folio may become anon once faulted in.
+ * File-backed order-1 folios are supported, since they do not use
+ * _deferred_list.
*
* After splitting, the caller's folio reference will be transferred to @page,
* resulting in a raised refcount of @page after this call. The other pages may
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 16/17] mm/huge_memory: clarify supported split orders in comment
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
The doc comment for __split_huge_page_to_list_to_order() needs an
update: only order 1 is rejected for anon and swapcache folios,
matching the new_order == 1 check in folio_check_splittable().
Also realign the continuation line of the function signature while at
it.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 317e5b63d44b..c8ec12f6f471 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4406,11 +4406,10 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
* GUP pins, will result in the folio not getting split; instead, the caller
* will receive an -EAGAIN.
*
- * 4) @new_order > 1, usually. Order-1 is not supported for anon or swapcache
- * folios: anon folios need subpage 2 for _deferred_list, which order-1
- * folios lack, and a swapcache folio may become anon once faulted in.
- * File-backed order-1 folios are supported, since they do not use
- * _deferred_list.
+ * 4) @new_order != 1 for anon or swapcache. Anon folios need subpage 2 for
+ * _deferred_list, which order-1 folios lack, and a swapcache folio may
+ * become anon once faulted in. File-backed order-1 folios are supported,
+ * since they do not use _deferred_list.
*
* After splitting, the caller's folio reference will be transferred to @page,
* resulting in a raised refcount of @page after this call. The other pages may
@@ -4438,7 +4437,7 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
* with the folio. Splitting to order 0 is compatible with all folios.
*/
int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
- unsigned int new_order)
+ unsigned int new_order)
{
struct folio *folio = page_folio(page);
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 16/17] mm/huge_memory: clarify supported split orders in comment
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
The doc comment for __split_huge_page_to_list_to_order() needs an
update: only order 1 is rejected for anon and swapcache folios,
matching the new_order == 1 check in folio_check_splittable().
Also realign the continuation line of the function signature while at
it.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 317e5b63d44b..c8ec12f6f471 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4406,11 +4406,10 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
* GUP pins, will result in the folio not getting split; instead, the caller
* will receive an -EAGAIN.
*
- * 4) @new_order > 1, usually. Order-1 is not supported for anon or swapcache
- * folios: anon folios need subpage 2 for _deferred_list, which order-1
- * folios lack, and a swapcache folio may become anon once faulted in.
- * File-backed order-1 folios are supported, since they do not use
- * _deferred_list.
+ * 4) @new_order != 1 for anon or swapcache. Anon folios need subpage 2 for
+ * _deferred_list, which order-1 folios lack, and a swapcache folio may
+ * become anon once faulted in. File-backed order-1 folios are supported,
+ * since they do not use _deferred_list.
*
* After splitting, the caller's folio reference will be transferred to @page,
* resulting in a raised refcount of @page after this call. The other pages may
@@ -4438,7 +4437,7 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
* with the folio. Splitting to order 0 is compatible with all folios.
*/
int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
- unsigned int new_order)
+ unsigned int new_order)
{
struct folio *folio = page_folio(page);
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 17/17] mm/huge_memory: count only swap cache refs in anon folio split
2026-08-12 18:48 ` Kairui Song
@ 2026-08-12 18:48 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song via B4 Relay @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
From: Kairui Song <kasong@tencent.com>
Only __folio_freeze_split_unmap() sees anon folios and swap cache folios
now. The file split helper only handles page cache folios, which hold
exactly folio_nr_pages() references.
Rename folio_cache_ref_count() to folio_swapcache_ref_count() and drop
the anon check so the helper counts what its name says. The file split
helper now uses folio_nr_pages() directly.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 35 +++++++++++++++--------------------
1 file changed, 15 insertions(+), 20 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c8ec12f6f471..1c61b7d39cd0 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3927,10 +3927,10 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
return 0;
}
-/* Number of folio references from the pagecache or the swapcache. */
-static unsigned int folio_cache_ref_count(const struct folio *folio)
+/* Number of folio references from the swapcache. */
+static unsigned int folio_swapcache_ref_count(const struct folio *folio)
{
- if (folio_test_anon(folio) && !folio_test_swapcache(folio))
+ if (!folio_test_swapcache(folio))
return 0;
return folio_nr_pages(folio);
}
@@ -4015,7 +4015,7 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
folio_nid(folio), &memcg);
}
- if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
+ if (!folio_ref_freeze(folio, folio_swapcache_ref_count(folio) + 1)) {
if (dequeue_deferred) {
list_lru_unlock(lru);
rcu_read_unlock();
@@ -4055,7 +4055,7 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
for_each_folio_safe(folio_next(folio), end_folio, new_folio, next) {
zone_device_private_split_cb(folio, new_folio);
folio_ref_unfreeze(new_folio,
- folio_cache_ref_count(new_folio) + 1);
+ folio_swapcache_ref_count(new_folio) + 1);
if (do_lru)
lru_add_split_folio(folio, new_folio, lruvec, list);
if (ci)
@@ -4063,7 +4063,7 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
}
zone_device_private_split_cb(folio, NULL);
- folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+ folio_ref_unfreeze(folio, folio_swapcache_ref_count(folio) + 1);
if (do_lru)
lruvec_unlock(lruvec);
@@ -4109,6 +4109,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
struct address_space *mapping = folio->mapping;
XA_STATE(xas, &mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
+ long old_nr_pages = folio_nr_pages(folio);
struct mem_cgroup *memcg, *old_memcg;
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
@@ -4180,22 +4181,16 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
goto fail;
}
- if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
+ if (!folio_ref_freeze(folio, old_nr_pages + 1)) {
ret = -EAGAIN;
goto fail;
}
- if (folio_test_pmd_mappable(folio) &&
- new_order < HPAGE_PMD_ORDER) {
- int nr = folio_nr_pages(folio);
-
- if (folio_test_swapbacked(folio)) {
- lruvec_stat_mod_folio(folio,
- NR_SHMEM_THPS, -nr);
- } else {
- lruvec_stat_mod_folio(folio,
- NR_FILE_THPS, -nr);
- }
+ if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) {
+ if (folio_test_swapbacked(folio))
+ lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -old_nr_pages);
+ else
+ lruvec_stat_mod_folio(folio, NR_FILE_THPS, -old_nr_pages);
}
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
@@ -4215,7 +4210,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
unsigned long nr_pages = folio_nr_pages(new_folio);
folio_ref_unfreeze(new_folio,
- folio_cache_ref_count(new_folio) + 1);
+ folio_nr_pages(new_folio) + 1);
lru_add_split_folio(folio, new_folio, lruvec, list);
@@ -4242,7 +4237,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
* Otherwise, a parallel folio_try_get() can grab @folio
* and its caller can see stale page cache entries.
*/
- folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+ folio_ref_unfreeze(folio, folio_nr_pages(folio) + 1);
lruvec_unlock(lruvec);
fail:
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH v2 17/17] mm/huge_memory: count only swap cache refs in anon folio split
@ 2026-08-12 18:48 ` Kairui Song
0 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-12 18:48 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra, Kairui Song
Only __folio_freeze_split_unmap() sees anon folios and swap cache folios
now. The file split helper only handles page cache folios, which hold
exactly folio_nr_pages() references.
Rename folio_cache_ref_count() to folio_swapcache_ref_count() and drop
the anon check so the helper counts what its name says. The file split
helper now uses folio_nr_pages() directly.
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 35 +++++++++++++++--------------------
1 file changed, 15 insertions(+), 20 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c8ec12f6f471..1c61b7d39cd0 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3927,10 +3927,10 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
return 0;
}
-/* Number of folio references from the pagecache or the swapcache. */
-static unsigned int folio_cache_ref_count(const struct folio *folio)
+/* Number of folio references from the swapcache. */
+static unsigned int folio_swapcache_ref_count(const struct folio *folio)
{
- if (folio_test_anon(folio) && !folio_test_swapcache(folio))
+ if (!folio_test_swapcache(folio))
return 0;
return folio_nr_pages(folio);
}
@@ -4015,7 +4015,7 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
folio_nid(folio), &memcg);
}
- if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
+ if (!folio_ref_freeze(folio, folio_swapcache_ref_count(folio) + 1)) {
if (dequeue_deferred) {
list_lru_unlock(lru);
rcu_read_unlock();
@@ -4055,7 +4055,7 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
for_each_folio_safe(folio_next(folio), end_folio, new_folio, next) {
zone_device_private_split_cb(folio, new_folio);
folio_ref_unfreeze(new_folio,
- folio_cache_ref_count(new_folio) + 1);
+ folio_swapcache_ref_count(new_folio) + 1);
if (do_lru)
lru_add_split_folio(folio, new_folio, lruvec, list);
if (ci)
@@ -4063,7 +4063,7 @@ static int __folio_split_unmap_and_freeze(struct folio *folio, unsigned int new_
}
zone_device_private_split_cb(folio, NULL);
- folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+ folio_ref_unfreeze(folio, folio_swapcache_ref_count(folio) + 1);
if (do_lru)
lruvec_unlock(lruvec);
@@ -4109,6 +4109,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
struct address_space *mapping = folio->mapping;
XA_STATE(xas, &mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
+ long old_nr_pages = folio_nr_pages(folio);
struct mem_cgroup *memcg, *old_memcg;
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
@@ -4180,22 +4181,16 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
goto fail;
}
- if (!folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
+ if (!folio_ref_freeze(folio, old_nr_pages + 1)) {
ret = -EAGAIN;
goto fail;
}
- if (folio_test_pmd_mappable(folio) &&
- new_order < HPAGE_PMD_ORDER) {
- int nr = folio_nr_pages(folio);
-
- if (folio_test_swapbacked(folio)) {
- lruvec_stat_mod_folio(folio,
- NR_SHMEM_THPS, -nr);
- } else {
- lruvec_stat_mod_folio(folio,
- NR_FILE_THPS, -nr);
- }
+ if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) {
+ if (folio_test_swapbacked(folio))
+ lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -old_nr_pages);
+ else
+ lruvec_stat_mod_folio(folio, NR_FILE_THPS, -old_nr_pages);
}
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
@@ -4215,7 +4210,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
unsigned long nr_pages = folio_nr_pages(new_folio);
folio_ref_unfreeze(new_folio,
- folio_cache_ref_count(new_folio) + 1);
+ folio_nr_pages(new_folio) + 1);
lru_add_split_folio(folio, new_folio, lruvec, list);
@@ -4242,7 +4237,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
* Otherwise, a parallel folio_try_get() can grab @folio
* and its caller can see stale page cache entries.
*/
- folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
+ folio_ref_unfreeze(folio, folio_nr_pages(folio) + 1);
lruvec_unlock(lruvec);
fail:
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [PATCH v2 02/17] mm/huge_memory: fix rejection of swap cache folios with a mapping
2026-08-12 18:48 ` Kairui Song
(?)
@ 2026-08-13 16:54 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-13 16:54 UTC (permalink / raw)
To: kasong
Cc: linux-mm, linux-kernel, Andrew Morton, David Hildenbrand,
Lorenzo Stoakes, Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache,
Ryan Roberts, Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Chris Li,
Kemeng Shi, Nhat Pham, Baoquan He, Barry Song, Youngjun Park,
Shivam Kalra
So sashiko reports (TLDR: false positive)
--- Patch [2]: [PATCH v2 02/17] mm/huge_memory: fix rejection of swap
cache folios with a mapping ---
commit de15de8c8cd6208f74b7fc4e795437aa367a9b26
Author: Kairui Song <kasong@tencent.com>
mm/huge_memory: fix rejection of swap cache folios with a mapping
A folio in the swap cache cannot be split if it has a mapping (shmem). The
split code currently does a defensive check for this after the folio ref
has been frozen, leaving it frozen on failure. This patch moves the check
up front to folio_check_splittable to reject the split before modifying
any state.
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index ced400f72d43a..a6759a14e057f 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
[ ... ]
> @@ -3983,14 +3994,8 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
> }
> }
>
> - if (folio_test_swapcache(folio)) {
> - if (mapping) {
> - VM_WARN_ON_ONCE_FOLIO(mapping, folio);
> - return -EINVAL;
> - }
> -
> + if (folio_test_swapcache(folio))
> ci = swap_cluster_get_and_lock(folio);
This is a pre-existing issue, but does acquiring spinlocks here while
hardware interrupts are disabled cause a sleeping-in-atomic-context BUG
on PREEMPT_RT?
Looking at __folio_split_unmap_and_freeze(), it explicitly disables
hardware interrupts earlier in the function:
local_irq_disable();
and then proceeds to acquire multiple spinlocks, such as in
swap_cluster_get_and_lock() and folio_lruvec_lock():
if (folio_test_swapcache(folio))
ci = swap_cluster_get_and_lock(folio);
if (do_lru)
lruvec = folio_lruvec_lock(folio);
On PREEMPT_RT kernels, spinlocks are mapped to sleepable rt_mutexes.
Attempting to acquire them after local_irq_disable() attempts to sleep
in atomic context, which can lead to system crashes or deadlocks when
large folios are split.
> /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
> if (do_lru)
---
It's pre-exsiting, and THP is not supported on RT yet, so no problem,
and that is exactly a good reason for the following patch to swtich
from local_irq_disable + spin_lock to spin_lock_irq. So we are in the
right direction :), and we can extend that further based on this
series.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 14/17] mm/huge_memory: clean up after-split folio freeing in __folio_split
2026-08-12 18:48 ` Kairui Song
(?)
@ 2026-08-13 17:04 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-13 17:04 UTC (permalink / raw)
To: kasong
Cc: linux-mm, linux-kernel, Andrew Morton, David Hildenbrand,
Lorenzo Stoakes, Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache,
Ryan Roberts, Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Chris Li,
Kemeng Shi, Nhat Pham, Baoquan He, Barry Song, Youngjun Park,
Shivam Kalra
Sashiko reports:
--- Patch [14]: [PATCH v2 14/17] mm/huge_memory: clean up after-split
folio freeing in __folio_split ---
commit 30068a849a3b68568f1661083bd4c5a685ac749a
Author: Kairui Song <kasong@tencent.com>
mm/huge_memory: clean up after-split folio freeing in __folio_split
This commit replaces free_folio_and_swap_cache() with direct calls to
folio_free_swap() and folio_put() in the __folio_split() after-split loop.
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -4343,14 +4343,16 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
> if (new_folio == page_folio(lock_at))
> continue;
>
> - folio_unlock(new_folio);
> /*
> * Subpages whose mapping has been zapped may be freed
> * earlier, but freeing them requires taking the
> - * lru_lock, so we defer put_page() on tail pages until
> + * lru_lock, so we defer folio_put() on tail pages until
> * after the split completes.
> */
> - free_folio_and_swap_cache(new_folio);
> + if (is_swapcache)
> + folio_free_swap(new_folio);
Is it safe to unconditionally call folio_free_swap() here without checking
if the subfolios are mapped?
The commit message states "The sub folios are still locked and unmapped
here", but looking at the earlier split path, __folio_split_unmap_and_freeze()
calls remap_page() before returning to __folio_split(). This means the
subfolios can actually be mapped in memory by the time we reach this point.
Because they are remapped, calling folio_free_swap() here will unconditionally
free the swap cache for actively mapped pages, marking them dirty and forcing
unnecessary swap allocations and disk writebacks under memory pressure.
Previously, free_folio_and_swap_cache() called free_swap_cache(), which
explicitly checked !folio_mapped() before freeing the cache to prevent this.
Does bypassing this check introduce a performance regression by thrashing
the swap cache for successfully split mapped anonymous folios?
> + folio_unlock(new_folio);
> + folio_put(new_folio);
> }
>
> out:
---
Right, I need to check the mapped status. It might be actually fine to
blindly clean the swap cache, and it's always safe to do so, but let
me keep the old behavior for now. I will update it.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 13/17] mm/huge_memory: drop the unused do_lru argument of the file split helper
2026-08-12 18:48 ` Kairui Song
(?)
@ 2026-08-13 18:02 ` Kairui Song
-1 siblings, 0 replies; 45+ messages in thread
From: Kairui Song @ 2026-08-13 18:02 UTC (permalink / raw)
To: kasong
Cc: linux-mm, linux-kernel, Andrew Morton, David Hildenbrand,
Lorenzo Stoakes, Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache,
Ryan Roberts, Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Chris Li,
Kemeng Shi, Nhat Pham, Baoquan He, Barry Song, Youngjun Park,
Shivam Kalra
Sashiko reports (TLDR: false positives):
--- Patch [13]: [PATCH v2 13/17] mm/huge_memory: drop the unused
do_lru argument of the file split helper ---
commit 56c82d4146d59b581770011278661a173c988702
Author: Kairui Song <kasong@tencent.com>
mm/huge_memory: drop the unused do_lru argument of the file split helper
The only caller of __folio_split_unmap_and_freeze_file() always passes
do_lru as true, so the argument and the branches gated on it are dead
code. Drop it.
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index ce02608b37f43..72e7d24139e6f 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
[ ... ]
> @@ -4206,9 +4204,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
> }
>
> /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
> - if (do_lru)
> - lruvec = folio_lruvec_lock(folio);
> -
> + lruvec = folio_lruvec_lock(folio);
> ret = __split_frozen_folio(folio, new_order, split_at, &xas,
> mapping, split_type);
This is a pre-existing issue, but does this code permanently leak NR_FILE_THPS
or NR_SHMEM_THPS stats if __split_frozen_folio() fails?
Right before this lock acquisition in __folio_split_unmap_and_freeze_file(),
the stats are eagerly decremented. If __split_frozen_folio() returns an error
(like -ENOMEM from xas_try_split() during a non-uniform split), the error
path fails to restore these stats.
Since the split failed, the folio remains pmd_mappable(). When the folio
is later evicted or freed, filemap_unaccount_folio() will observe it is still
pmd_mappable() and decrement the stats again, causing a double-decrement
underflow.
[ ... ]
> @@ -4226,8 +4222,7 @@ static int __folio_split_unmap_and_freeze_file(struct folio *folio, unsigned int
> folio_ref_unfreeze(new_folio,
> folio_cache_ref_count(new_folio) + 1);
>
> - if (do_lru)
> - lru_add_split_folio(folio, new_folio, lruvec, list);
> + lru_add_split_folio(folio, new_folio, lruvec, list);
>
> /* Add the new folio to the page cache. */
> if (new_folio->index < end) {
This is a pre-existing issue, but does calling __xa_store() here inside
the page cache insertion loop leak memory and cause latency spikes?
__folio_split_unmap_and_freeze_file() correctly sets up a local XA_STATE
with mapping_set_update(&xas, mapping). However, the page cache insertion
loop (just below this snippet) uses __xa_store(), which creates a fresh local
XA_STATE that lacks the workingset_update_node callback.
Without this callback, xa_nodes are not added to their memcg's list_lru,
leaking nodes under memory pressure.
Additionally, because __xa_store() bypasses the advanced XA_STATE and
restarts from the root for every sub-folio, it turns an O(1) advancement
into an O(log N) tree walk for each slot while holding the xas_lock_irq
spinlock, which can cause severe latency.
---
For the stats question: the decrement is done up front, but no restore is
needed on the error path. The uniform path preallocates all the xarray
nodes and checks for errors before the stats are touched, so it cannot
fail there. The non-uniform path can fail - xas_try_split() does one
GFP_NOWAIT allocation when reducing the order-6 chunk - but by then the
folio has already been split below PMD order, so a failed split can only
leave behind an order-6 remnant, which is not pmd sized.
For the __xa_store() one: it doesn't need to allocate nodes,
every slot it writes already exists after xas_split()/xas_try_split().
Nodes the split creates are tracked through
workingset_update_node() (mapping_set_update() is set on the xas), so
nothing leaks. The fresh root walk per sub-folio is real, not a bug,
It may be worth optimizing later.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 04/17] mm/huge_memory: split the routine for splitting anon and file folio
2026-08-12 18:48 ` Kairui Song
(?)
@ 2026-08-13 18:11 ` Kairui Song
2026-08-15 2:37 ` Zi Yan
-1 siblings, 1 reply; 45+ messages in thread
From: Kairui Song @ 2026-08-13 18:11 UTC (permalink / raw)
To: kasong
Cc: linux-mm, linux-kernel, Andrew Morton, David Hildenbrand,
Lorenzo Stoakes, Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache,
Ryan Roberts, Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Chris Li,
Kemeng Shi, Nhat Pham, Baoquan He, Barry Song, Youngjun Park,
Shivam Kalra
Sashiko reports (a valid case this time):
--- Patch [4]: [PATCH v2 04/17] mm/huge_memory: split the routine for
splitting anon and file folio ---
commit 249f031b2182770a8cf754edfb3833e148a033df
Author: Kairui Song <kasong@tencent.com>
mm/huge_memory: split the routine for splitting anon and file folio
No functional change intended. Before adding more logic, split
__folio_freeze_and_split_unmapped() into an anon and a file variant so
each path can evolve independently. The two paths shared little beyond
the folio freeze call, the LRU locking, and the unfreeze skeleton, but
differed in all other per-folio bookkeeping and routines.
While splitting, some cleanups become easy to apply, and helped drop a
few now-redundant checks. Also introduce a folio iteration helper to
avoid a common pitfall of iterating post-split sub-folios: a sub folio
might get freed mid-iteration as pointed out by Zi [1].
Link: https://lore.kernel.org/linux-mm/DKJSFCLP967N.YBR4DNK1NM2N@nvidia.com/ [1]
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 7fb603ac500f6..7587eeb09e4a6 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
[ ... ]
> @@ -3634,6 +3634,18 @@ static bool page_range_has_hwpoisoned(struct page *page, long nr_pages)
> return false;
> }
>
> +/**
> + * for_each_folio_safe - iterate over contiguous folios safe against folio free
> + * @start: the first folio to iterate
> + * @end: sentinel, folio_next() of the last folio to iterate
> + * @sub_folio: struct folio * to use as the loop cursor
> + * @next: struct folio * used as temporary storage
> + */
> +#define for_each_folio_safe(start, end, sub_folio, next) \
> + for (sub_folio = (start), next = folio_next(sub_folio); \
> + sub_folio != (end); \
> + sub_folio = next, next = folio_next(next))
Does this macro unconditionally dereference the sentinel pointer?
When the loop increment step executes, it evaluates both sub_folio = next
and next = folio_next(next). If next has reached the end sentinel, calling
folio_next(next) will dereference the sentinel pointer to read its metadata.
If the split large folio happens to be located at the end of a populated
memory section, the sentinel pointer (end) could point to the next section's
vmemmap. If that next section is unpopulated, this out-of-bounds read will
trigger a fatal kernel page fault.
---
Right, I actually hesitated on that but forgot about it midway. I
think I'll just change back to the opencoded style, it's easier to
understand that way.
Or does anyone like this (copied from hlist_for_each_safe)?
#define for_each_folio_safe(start, end, sub_folio, next) \
for (sub_folio = (start); \
sub_folio != (end) && ({ next = folio_next(sub_folio); 1; }); \
sub_folio = next)
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 00/17] mm/huge_memory: clean up folio split and lift swapcache split limits
2026-08-12 18:48 ` Kairui Song
` (17 preceding siblings ...)
(?)
@ 2026-08-13 19:35 ` David Hildenbrand (Arm)
-1 siblings, 0 replies; 45+ messages in thread
From: David Hildenbrand (Arm) @ 2026-08-13 19:35 UTC (permalink / raw)
To: kasong, linux-mm
Cc: linux-kernel, Andrew Morton, Lorenzo Stoakes, Zi Yan, Baolin Wang,
Liam R. Howlett, Nico Pache, Ryan Roberts, Dev Jain, Lance Yang,
Usama Arif, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan,
Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Barry Song, Youngjun Park, Shivam Kalra
On 8/12/26 20:48, Kairui Song via B4 Relay wrote:
> This series clean up the split code, add better swap cache split support
> for mappingless, large order, uniform and non-uniform split. Generic
> performance is on par or slightly better, and stack usage is reduced.
>
> The swap cache infrastructure can handle non-uniform or high order folio
> replace, so there is no reason for either restriction from the THP side.
> What stands in the way is the mixed anon/file folio split routine,
> which makes lifting the restrictions hard to follow, and it already
> carries some buggy or redundant checks.
>
> So this series cleans up the split path and separates anon and file
> splitting into two helpers. The file split path never sees a swap
> cache folio, and that is now enforced up front: a folio that is both
> in the page cache and the swap cache can only be a shmem folio, which
> remains unsupported and is rejected early. That helps to rule out swap
> cache handling in that part completely. Only the anon split path
> handles swap cache folios, with an anon mapping or mappingless:
> either way the splitting is similar, and non-uniform split is
> supported as well.
>
> Order-1 is still forbidden for swap cache splitting. In theory it is
> doable for shmem swap cache folios, but a mappingless swap cache
> folio cannot currently be told apart from a shmem one, so forbid it
> for all swap cache folios for now.
>
> Testing:
>
> The in-tree split_huge_page_test selftest (uniform, non-uniform and
> in-folio-offset splits of anon and pagecache folios) passes 62/62 on
> the patched kernel.
>
> ftrace function_graph tracing filtered on __folio_split() was used to
> compare per-call durations between the base and the patched kernel on
> the same x86-64 box (interleaved runs across alternating reboots;
> mean +- stddev of the per-run averages, 135 split calls per run):
>
> base: 24 runs, 69.6 +- 0.7 us per __folio_split()
> patched: 26 runs, 68.8 +- 1.3 us per __folio_split()
>
> The patched kernel is consistently ~1% faster; with this sample
> count the difference is outside run-to-run noise.
>
> On x86-64 with gcc 12 (-fstack-usage), the stack frame of
> __folio_split() shrinks from 240 to 96 bytes, and the worst-case
> split call chain from ~544 to ~384 (anon) or ~464 (file) bytes.
>
> Bloat-o-meter shows a tiny growth of huge_memory.o:
> before=58419 after=58446, chg +0.05% (+27 bytes).
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
I might need a bit to get to this; but the merge window is about to open either
way so, so this is material for the one afterwards.
--
Cheers,
David
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 02/17] mm/huge_memory: fix rejection of swap cache folios with a mapping
2026-08-12 18:48 ` Kairui Song
(?)
(?)
@ 2026-08-15 2:25 ` Zi Yan
-1 siblings, 0 replies; 45+ messages in thread
From: Zi Yan @ 2026-08-15 2:25 UTC (permalink / raw)
To: kasong, linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts, Dev Jain,
Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra
On Wed Aug 12, 2026 at 2:48 PM EDT, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> A folio in the swap cache cannot be split if it has a mapping (shmem).
> The split code does a defensive check for this in
> __folio_freeze_and_split_unmapped, after the folio ref has been frozen
> and the NR_SHMEM_THPS/NR_FILE_THPS counters have been decremented. It
> rejects the split and returns -EINVAL without unfreezing the folio or
> restoring the counters. That error path is buggy: if it is ever taken,
> it leaves the folio frozen and stuck, skews the counters, and fires
> the VM_WARN_ON_ONCE_FOLIO for a state that is actually legitimate.
>
> Check for this case up front in folio_check_splittable and return
> -EBUSY before any state is modified, so the split routine always backs
> out cleanly.
>
> Also fix a bracket style issue that checkpatch.pl keeps complaining
> about.
>
> Fixes: 00527733d0dc ("mm/huge_memory: add two new (not yet used) functions for folio_split()")
> Fixes: 714b056c8321 ("mm/huge_memory: convert VM_BUG* to VM_WARN* in __folio_split")
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/huge_memory.c | 27 ++++++++++++++++-----------
> 1 file changed, 16 insertions(+), 11 deletions(-)
>
The handling is an improvement. Thanks.
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 04/17] mm/huge_memory: split the routine for splitting anon and file folio
2026-08-13 18:11 ` Kairui Song
@ 2026-08-15 2:37 ` Zi Yan
0 siblings, 0 replies; 45+ messages in thread
From: Zi Yan @ 2026-08-15 2:37 UTC (permalink / raw)
To: Kairui Song, kasong
Cc: linux-mm, linux-kernel, Andrew Morton, David Hildenbrand,
Lorenzo Stoakes, Baolin Wang, Liam R. Howlett, Nico Pache,
Ryan Roberts, Dev Jain, Lance Yang, Usama Arif, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Chris Li,
Kemeng Shi, Nhat Pham, Baoquan He, Barry Song, Youngjun Park,
Shivam Kalra
On Thu Aug 13, 2026 at 2:11 PM EDT, Kairui Song wrote:
> Sashiko reports (a valid case this time):
>
> --- Patch [4]: [PATCH v2 04/17] mm/huge_memory: split the routine for
> splitting anon and file folio ---
> commit 249f031b2182770a8cf754edfb3833e148a033df
> Author: Kairui Song <kasong@tencent.com>
>
> mm/huge_memory: split the routine for splitting anon and file folio
>
> No functional change intended. Before adding more logic, split
> __folio_freeze_and_split_unmapped() into an anon and a file variant so
> each path can evolve independently. The two paths shared little beyond
> the folio freeze call, the LRU locking, and the unfreeze skeleton, but
> differed in all other per-folio bookkeeping and routines.
>
> While splitting, some cleanups become easy to apply, and helped drop a
> few now-redundant checks. Also introduce a folio iteration helper to
> avoid a common pitfall of iterating post-split sub-folios: a sub folio
> might get freed mid-iteration as pointed out by Zi [1].
>
> Link: https://lore.kernel.org/linux-mm/DKJSFCLP967N.YBR4DNK1NM2N@nvidia.com/ [1]
>
>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
>> index 7fb603ac500f6..7587eeb09e4a6 100644
>> --- a/mm/huge_memory.c
>> +++ b/mm/huge_memory.c
>
> [ ... ]
>
>> @@ -3634,6 +3634,18 @@ static bool page_range_has_hwpoisoned(struct page *page, long nr_pages)
>> return false;
>> }
>>
>> +/**
>> + * for_each_folio_safe - iterate over contiguous folios safe against folio free
>> + * @start: the first folio to iterate
>> + * @end: sentinel, folio_next() of the last folio to iterate
>> + * @sub_folio: struct folio * to use as the loop cursor
>> + * @next: struct folio * used as temporary storage
>> + */
>> +#define for_each_folio_safe(start, end, sub_folio, next) \
>> + for (sub_folio = (start), next = folio_next(sub_folio); \
>> + sub_folio != (end); \
>> + sub_folio = next, next = folio_next(next))
>
> Does this macro unconditionally dereference the sentinel pointer?
>
> When the loop increment step executes, it evaluates both sub_folio = next
> and next = folio_next(next). If next has reached the end sentinel, calling
> folio_next(next) will dereference the sentinel pointer to read its metadata.
>
> If the split large folio happens to be located at the end of a populated
> memory section, the sentinel pointer (end) could point to the next section's
> vmemmap. If that next section is unpopulated, this out-of-bounds read will
> trigger a fatal kernel page fault.
>
> ---
>
> Right, I actually hesitated on that but forgot about it midway. I
> think I'll just change back to the opencoded style, it's easier to
> understand that way.
Let's open code it. I also tried a similar macro when I was developing
folio_split(), but got convinced it buys nothing.
If the motivation comes my prior comment on code duplication, feel free
to ignore that. If we over engineer it, like different function pointers
for anon/swapcache and file, it might be doable. But that might kill
readability.
What I mean is something like:
__folio_freeze_and_split_unmapped(func_t pre_freeze_func, func_t
unfreeze_func, func_t post_freeze_func)
{
pre_freeze_func();
for () {
unfreeze_func();
}
post_freeze_func();
}
for anon/swapcache:
__folio_freeze_and_split_unmapped(anon_swapcache_pre_freeze,
anon_swapcache_unfreeze, anon_swapcache_post_freeze);
for file:
__folio_freeze_and_split_unmapped(file_pre_freeze, file_unfreeze,
file_post_freeze);
--
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 05/17] mm/huge_memory: rename __split_unmapped_folio() to __split_frozen_folio()
2026-08-12 18:48 ` Kairui Song
(?)
@ 2026-08-15 2:38 ` Zi Yan
-1 siblings, 0 replies; 45+ messages in thread
From: Zi Yan @ 2026-08-15 2:38 UTC (permalink / raw)
To: kasong, linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts, Dev Jain,
Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra
On Wed Aug 12, 2026 at 2:48 PM EDT, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> The helper splits a folio whose refcount is frozen: the frozen refcount
> is the state it relies on, while unmapping is arranged by the caller
> beforehand. The old name caused confusion and people may try to call the
> helper on non-frozen folios.
>
> Suggested-by: Zi Yan <ziy@nvidia.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/huge_memory.c | 20 ++++++++++----------
> 1 file changed, 10 insertions(+), 10 deletions(-)
>
Thanks.
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH v2 06/17] mm/huge_memory: consolidate irq and locking for folio split
2026-08-12 18:48 ` Kairui Song
(?)
@ 2026-08-15 2:49 ` Zi Yan
-1 siblings, 0 replies; 45+ messages in thread
From: Zi Yan @ 2026-08-15 2:49 UTC (permalink / raw)
To: kasong, linux-mm
Cc: linux-kernel, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts, Dev Jain,
Lance Yang, Usama Arif, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Barry Song, Youngjun Park, Shivam Kalra
On Wed Aug 12, 2026 at 2:48 PM EDT, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> Let each split helper handle its own locking instead of relying on
> the caller, so both helpers manage their own irq and locking state.
> This lets __folio_split() drop its local irq handling and fail label,
> preparing for further cleanup.
>
> The file path now uses xas_lock_irq() instead of local_irq_disable()
> with xas_lock(). The two are equivalent on non-RT, and
> TRANSPARENT_HUGEPAGE cannot be enabled on RT anyway. This conversion
> also buys consistency: every other place in mm/ that freezes a folio
> while it is still reachable through the page cache already takes the
> lock this way. This was actually the last plain xas_lock() on
> mapping->i_pages left in mm. If we are going to support RT, spinning
> on frozen folio refs could be a problem, but it already exists in
> many places and should be fixed generically.
>
> The anon helper keeps a single local_irq_disable() as before, because
> it has to cover several plain spinlocks at once.
>
> The dropped xas_reset() was a no-op as the xa_state is not walked
> before the xas_load() under the lock.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/huge_memory.c | 52 ++++++++++++++++++++++++----------------------------
> 1 file changed, 24 insertions(+), 28 deletions(-)
>
The changes look good to me and the commit message looks great!
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 45+ messages in thread
end of thread, other threads:[~2026-08-15 2:49 UTC | newest]
Thread overview: 45+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-12 18:48 [PATCH v2 00/17] mm/huge_memory: clean up folio split and lift swapcache split limits Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 01/17] mm/swap: fix off-by-one in swap cache replace sanity check Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 02/17] mm/huge_memory: fix rejection of swap cache folios with a mapping Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-13 16:54 ` Kairui Song
2026-08-15 2:25 ` Zi Yan
2026-08-12 18:48 ` [PATCH v2 03/17] mm/huge_memory: invert folio_ref_freeze() check to reduce indentation Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 04/17] mm/huge_memory: split the routine for splitting anon and file folio Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-13 18:11 ` Kairui Song
2026-08-15 2:37 ` Zi Yan
2026-08-12 18:48 ` [PATCH v2 05/17] mm/huge_memory: rename __split_unmapped_folio() to __split_frozen_folio() Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-15 2:38 ` Zi Yan
2026-08-12 18:48 ` [PATCH v2 06/17] mm/huge_memory: consolidate irq and locking for folio split Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-15 2:49 ` Zi Yan
2026-08-12 18:48 ` [PATCH v2 07/17] mm/huge_memory: move EOF trimming into the file split helper Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 08/17] mm/huge_memory: move unmap and remap into the split helpers Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 09/17] mm/huge_memory: move anon_vma and filemap management into " Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 10/17] mm/huge_memory: move memcg switch into the file split helper Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 11/17] mm/huge_memory: allow splitting mappingless swap cache folios Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 12/17] mm/huge_memory: add kerneldoc for the split helpers Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 13/17] mm/huge_memory: drop the unused do_lru argument of the file split helper Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-13 18:02 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 14/17] mm/huge_memory: clean up after-split folio freeing in __folio_split Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-13 17:04 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 15/17] mm/huge_memory: lift order-0 restriction for swapcache split Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 16/17] mm/huge_memory: clarify supported split orders in comment Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-12 18:48 ` [PATCH v2 17/17] mm/huge_memory: count only swap cache refs in anon folio split Kairui Song via B4 Relay
2026-08-12 18:48 ` Kairui Song
2026-08-13 19:35 ` [PATCH v2 00/17] mm/huge_memory: clean up folio split and lift swapcache split limits David Hildenbrand (Arm)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.