* [PATCH 6.18.y 2/6] mm: thp: use folio_batch to handle THP splitting in deferred_split_scan()
2026-09-03 18:26 ` [PATCH 6.18.y 1/6] mm: thp: introduce folio_split_queue_lock and its variants Sasha Levin
@ 2026-09-03 18:26 ` Sasha Levin
2026-09-03 19:28 ` Hugh Dickins
2026-09-03 18:26 ` [PATCH 6.18.y 3/6] mm/huge_memory: change folio_split_supported() to folio_check_splittable() Sasha Levin
` (3 subsequent siblings)
4 siblings, 1 reply; 8+ messages in thread
From: Sasha Levin @ 2026-09-03 18:26 UTC (permalink / raw)
To: stable
Cc: Muchun Song, Qi Zheng, Zi Yan, David Hildenbrand, Shakeel Butt,
Wei Yang, Harry Yoo, Baolin Wang, Barry Song, Dev Jain,
Hugh Dickins, Johannes Weiner, Lance Yang, Liam Howlett,
Lorenzo Stoakes, Michal Hocko, Muchun Song, Nico Pache,
Roman Gushchin, Ryan Roberts, Andrew Morton, Sasha Levin
From: Muchun Song <songmuchun@bytedance.com>
[ Upstream commit 776bde7caf80f6af72b087cafe7d9f607b14716d ]
The maintenance of the folio->_deferred_list is intricate because it's
reused in a local list.
Here are some peculiarities:
1) When a folio is removed from its split queue and added to a local
on-stack list in deferred_split_scan(), the ->split_queue_len isn't
updated, leading to an inconsistency between it and the actual
number of folios in the split queue.
2) When the folio is split via split_folio() later, it's removed from
the local list while holding the split queue lock. At this time,
the lock is not needed as it is not protecting anything.
3) To handle the race condition with a third-party freeing or migrating
the preceding folio, we must ensure there's always one safe (with
raised refcount) folio before by delaying its folio_put(). More
details can be found in commit e66f3185fa04 ("mm/thp: fix deferred
split queue not partially_mapped"). It's rather tricky.
We can use the folio_batch infrastructure to handle this clearly. In this
case, ->split_queue_len will be consistent with the real number of folios
in the split queue. If list_empty(&folio->_deferred_list) returns false,
it's clear the folio must be in its split queue (not in a local list
anymore).
In the future, we will reparent LRU folios during memcg offline to
eliminate dying memory cgroups, which requires reparenting the split queue
to its parent first. So this patch prepares for using
folio_split_queue_lock_irqsave() as the memcg may change then.
Link: https://lkml.kernel.org/r/59cb6b6fb5ffcff9d23b81890b252960139ad8e7.1762762324.git.zhengqi.arch@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand <david@redhat.com>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nico Pache <npache@redhat.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
mm/huge_memory.c | 130 ++++++++++++++++++++++++-----------------------
1 file changed, 66 insertions(+), 64 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 11a58a93f361b..27d97afd22be1 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3686,19 +3686,21 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
bool is_anon = folio_test_anon(folio);
struct address_space *mapping = NULL;
struct anon_vma *anon_vma = NULL;
- int order = folio_order(folio);
+ int old_order = folio_order(folio);
struct folio *new_folio, *next;
+ int extra_pins;
int nr_shmem_dropped = 0;
- int remap_flags = 0;
- int extra_pins, ret;
- pgoff_t end;
- bool is_hzp;
+ enum ttu_flags ttu_flags = 0;
+ pgoff_t end = 0;
+ int ret;
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
- if (folio != page_folio(split_at) || folio != page_folio(lock_at))
- return -EINVAL;
+ if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
+ ret = -EINVAL;
+ goto out;
+ }
/*
* Folios that just got truncated cannot get split. Signal to the
@@ -3710,14 +3712,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (!is_anon && !folio->mapping)
return -EBUSY;
- if (new_order >= folio_order(folio))
- return -EINVAL;
-
- if (!folio_split_supported(folio, new_order, uniform_split, /* warn = */ true))
- return -EINVAL;
+ if (new_order >= old_order) {
+ ret = -EINVAL;
+ goto out;
+ }
- is_hzp = is_huge_zero_folio(folio);
- if (is_hzp) {
+ if (is_huge_zero_folio(folio)) {
pr_warn_ratelimited("Called split_huge_page for huge zero page\n");
return -EBUSY;
}
@@ -3725,6 +3725,13 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (folio_test_writeback(folio))
return -EBUSY;
+ ret = folio_split_supported(folio, new_order, uniform_split,
+ /* warns = */ false) ? 0 : -EINVAL;
+ if (ret) {
+ VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
+ goto out;
+ }
+
if (is_anon) {
/*
* The caller does not necessarily hold an mmap_lock that would
@@ -3817,21 +3824,22 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct lruvec *lruvec;
int expected_refs;
- if (folio_order(folio) > 1 &&
- !list_empty(&folio->_deferred_list)) {
- ds_queue->split_queue_len--;
+ if (folio_order(folio) > 1) {
+ if (!list_empty(&folio->_deferred_list)) {
+ ds_queue->split_queue_len--;
+ /*
+ * Reinitialize page_deferred_list after removing the
+ * page from the split_queue, otherwise a subsequent
+ * split will see list corruption when checking the
+ * page_deferred_list.
+ */
+ list_del_init(&folio->_deferred_list);
+ }
if (folio_test_partially_mapped(folio)) {
folio_clear_partially_mapped(folio);
mod_mthp_stat(folio_order(folio),
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
- /*
- * Reinitialize page_deferred_list after removing the
- * page from the split_queue, otherwise a subsequent
- * split will see list corruption when checking the
- * page_deferred_list.
- */
- list_del_init(&folio->_deferred_list);
}
split_queue_unlock(ds_queue);
if (mapping) {
@@ -3941,8 +3949,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
shmem_uncharge(mapping->host, nr_shmem_dropped);
if (!ret && is_anon)
- remap_flags = RMP_USE_SHARED_ZEROPAGE;
- remap_page(folio, 1 << order, remap_flags);
+ ttu_flags = (enum ttu_flags)RMP_USE_SHARED_ZEROPAGE;
+ remap_page(folio, 1 << old_order, ttu_flags);
/*
* Drop the mapping while the inode is still pinned. @folio stays
@@ -3985,9 +3993,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
i_mmap_unlock_read(mapping);
out:
xas_destroy(&xas);
- if (order == HPAGE_PMD_ORDER)
+ if (old_order == HPAGE_PMD_ORDER)
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
- count_mthp_stat(order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
+ count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
return ret;
}
@@ -4230,35 +4238,40 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
{
struct deferred_split *ds_queue;
unsigned long flags;
- LIST_HEAD(list);
- struct folio *folio, *next, *prev = NULL;
- int split = 0, removed = 0;
+ struct folio *folio, *next;
+ int split = 0, i;
+ struct folio_batch fbatch;
+
+ folio_batch_init(&fbatch);
+retry:
ds_queue = split_queue_lock_irqsave(sc->nid, sc->memcg, &flags);
/* Take pin on all head pages to avoid freeing them under us */
list_for_each_entry_safe(folio, next, &ds_queue->split_queue,
_deferred_list) {
if (folio_try_get(folio)) {
- list_move(&folio->_deferred_list, &list);
- } else {
+ folio_batch_add(&fbatch, folio);
+ } else if (folio_test_partially_mapped(folio)) {
/* We lost race with folio_put() */
- if (folio_test_partially_mapped(folio)) {
- folio_clear_partially_mapped(folio);
- mod_mthp_stat(folio_order(folio),
- MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
- }
- list_del_init(&folio->_deferred_list);
- ds_queue->split_queue_len--;
+ folio_clear_partially_mapped(folio);
+ mod_mthp_stat(folio_order(folio),
+ MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
+ list_del_init(&folio->_deferred_list);
+ ds_queue->split_queue_len--;
if (!--sc->nr_to_scan)
break;
+ if (!folio_batch_space(&fbatch))
+ break;
}
split_queue_unlock_irqrestore(ds_queue, flags);
- list_for_each_entry_safe(folio, next, &list, _deferred_list) {
+ for (i = 0; i < folio_batch_count(&fbatch); i++) {
bool did_split = false;
bool underused = false;
+ struct deferred_split *fqueue;
+ folio = fbatch.folios[i];
if (!folio_test_partially_mapped(folio)) {
/*
* See try_to_map_unused_to_zeropage(): we cannot
@@ -4281,38 +4294,27 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
}
folio_unlock(folio);
next:
+ if (did_split || !folio_test_partially_mapped(folio))
+ continue;
/*
- * split_folio() removes folio from list on success.
* Only add back to the queue if folio is partially mapped.
* If thp_underused returns false, or if split_folio fails
* in the case it was underused, then consider it used and
* don't add it back to split_queue.
*/
- if (did_split) {
- ; /* folio already removed from list */
- } else if (!folio_test_partially_mapped(folio)) {
- list_del_init(&folio->_deferred_list);
- removed++;
- } else {
- /*
- * That unlocked list_del_init() above would be unsafe,
- * unless its folio is separated from any earlier folios
- * left on the list (which may be concurrently unqueued)
- * by one safe folio with refcount still raised.
- */
- swap(folio, prev);
+ fqueue = folio_split_queue_lock_irqsave(folio, &flags);
+ if (list_empty(&folio->_deferred_list)) {
+ list_add_tail(&folio->_deferred_list, &fqueue->split_queue);
+ fqueue->split_queue_len++;
}
- if (folio)
- folio_put(folio);
+ split_queue_unlock_irqrestore(fqueue, flags);
}
+ folios_put(&fbatch);
- spin_lock_irqsave(&ds_queue->split_queue_lock, flags);
- list_splice_tail(&list, &ds_queue->split_queue);
- ds_queue->split_queue_len -= removed;
- spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags);
-
- if (prev)
- folio_put(prev);
+ if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) {
+ cond_resched();
+ goto retry;
+ }
/*
* Stop shrinker if we didn't split any page, but the queue is empty.
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH 6.18.y 2/6] mm: thp: use folio_batch to handle THP splitting in deferred_split_scan()
2026-09-03 18:26 ` [PATCH 6.18.y 2/6] mm: thp: use folio_batch to handle THP splitting in deferred_split_scan() Sasha Levin
@ 2026-09-03 19:28 ` Hugh Dickins
0 siblings, 0 replies; 8+ messages in thread
From: Hugh Dickins @ 2026-09-03 19:28 UTC (permalink / raw)
To: Sasha Levin
Cc: stable, Muchun Song, Qi Zheng, Zi Yan, David Hildenbrand,
Shakeel Butt, Wei Yang, Harry Yoo, Baolin Wang, Barry Song,
Dev Jain, Hugh Dickins, Johannes Weiner, Lance Yang, Liam Howlett,
Lorenzo Stoakes, Michal Hocko, Muchun Song, Nico Pache,
Roman Gushchin, Ryan Roberts, Andrew Morton
On Thu, 3 Sep 2026, Sasha Levin wrote:
> From: Muchun Song <songmuchun@bytedance.com>
>
> [ Upstream commit 776bde7caf80f6af72b087cafe7d9f607b14716d ]
>
> The maintenance of the folio->_deferred_list is intricate because it's
> reused in a local list.
>
> Here are some peculiarities:
>
> 1) When a folio is removed from its split queue and added to a local
> on-stack list in deferred_split_scan(), the ->split_queue_len isn't
> updated, leading to an inconsistency between it and the actual
> number of folios in the split queue.
>
> 2) When the folio is split via split_folio() later, it's removed from
> the local list while holding the split queue lock. At this time,
> the lock is not needed as it is not protecting anything.
>
> 3) To handle the race condition with a third-party freeing or migrating
> the preceding folio, we must ensure there's always one safe (with
> raised refcount) folio before by delaying its folio_put(). More
> details can be found in commit e66f3185fa04 ("mm/thp: fix deferred
> split queue not partially_mapped"). It's rather tricky.
>
> We can use the folio_batch infrastructure to handle this clearly. In this
> case, ->split_queue_len will be consistent with the real number of folios
> in the split queue. If list_empty(&folio->_deferred_list) returns false,
> it's clear the folio must be in its split queue (not in a local list
> anymore).
>
> In the future, we will reparent LRU folios during memcg offline to
> eliminate dying memory cgroups, which requires reparenting the split queue
> to its parent first. So this patch prepares for using
> folio_split_queue_lock_irqsave() as the memcg may change then.
>
> Link: https://lkml.kernel.org/r/59cb6b6fb5ffcff9d23b81890b252960139ad8e7.1762762324.git.zhengqi.arch@bytedance.com
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Acked-by: David Hildenbrand <david@redhat.com>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
> Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
> Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
> Cc: Barry Song <baohua@kernel.org>
> Cc: Dev Jain <dev.jain@arm.com>
> Cc: Hugh Dickins <hughd@google.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Lance Yang <lance.yang@linux.dev>
> Cc: Liam Howlett <liam.howlett@oracle.com>
> Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Muchun Song <muchun.song@linux.dev>
> Cc: Nico Pache <npache@redhat.com>
> Cc: Roman Gushchin <roman.gushchin@linux.dev>
> Cc: Ryan Roberts <ryan.roberts@arm.com>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
> Signed-off-by: Sasha Levin <sashal@kernel.org>
Oh dear, more of these Stable-dep-ofs. I just happened to be Cc'ed on
this one, and after commenting on the pincount ones, I'd better comment
on this too: we definitely want Zi Yan's XA_FLAGS_ACCOUNT fix, but
deferred split queue is tricky: so I'd judge that this should not
go forward without explicit Acks from... whoever knows best.
Hugh
> ---
> mm/huge_memory.c | 130 ++++++++++++++++++++++++-----------------------
> 1 file changed, 66 insertions(+), 64 deletions(-)
>
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 11a58a93f361b..27d97afd22be1 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -3686,19 +3686,21 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
> bool is_anon = folio_test_anon(folio);
> struct address_space *mapping = NULL;
> struct anon_vma *anon_vma = NULL;
> - int order = folio_order(folio);
> + int old_order = folio_order(folio);
> struct folio *new_folio, *next;
> + int extra_pins;
> int nr_shmem_dropped = 0;
> - int remap_flags = 0;
> - int extra_pins, ret;
> - pgoff_t end;
> - bool is_hzp;
> + enum ttu_flags ttu_flags = 0;
> + pgoff_t end = 0;
> + int ret;
>
> VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
> VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
>
> - if (folio != page_folio(split_at) || folio != page_folio(lock_at))
> - return -EINVAL;
> + if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
> + ret = -EINVAL;
> + goto out;
> + }
>
> /*
> * Folios that just got truncated cannot get split. Signal to the
> @@ -3710,14 +3712,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
> if (!is_anon && !folio->mapping)
> return -EBUSY;
>
> - if (new_order >= folio_order(folio))
> - return -EINVAL;
> -
> - if (!folio_split_supported(folio, new_order, uniform_split, /* warn = */ true))
> - return -EINVAL;
> + if (new_order >= old_order) {
> + ret = -EINVAL;
> + goto out;
> + }
>
> - is_hzp = is_huge_zero_folio(folio);
> - if (is_hzp) {
> + if (is_huge_zero_folio(folio)) {
> pr_warn_ratelimited("Called split_huge_page for huge zero page\n");
> return -EBUSY;
> }
> @@ -3725,6 +3725,13 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
> if (folio_test_writeback(folio))
> return -EBUSY;
>
> + ret = folio_split_supported(folio, new_order, uniform_split,
> + /* warns = */ false) ? 0 : -EINVAL;
> + if (ret) {
> + VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
> + goto out;
> + }
> +
> if (is_anon) {
> /*
> * The caller does not necessarily hold an mmap_lock that would
> @@ -3817,21 +3824,22 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
> struct lruvec *lruvec;
> int expected_refs;
>
> - if (folio_order(folio) > 1 &&
> - !list_empty(&folio->_deferred_list)) {
> - ds_queue->split_queue_len--;
> + if (folio_order(folio) > 1) {
> + if (!list_empty(&folio->_deferred_list)) {
> + ds_queue->split_queue_len--;
> + /*
> + * Reinitialize page_deferred_list after removing the
> + * page from the split_queue, otherwise a subsequent
> + * split will see list corruption when checking the
> + * page_deferred_list.
> + */
> + list_del_init(&folio->_deferred_list);
> + }
> if (folio_test_partially_mapped(folio)) {
> folio_clear_partially_mapped(folio);
> mod_mthp_stat(folio_order(folio),
> MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
> }
> - /*
> - * Reinitialize page_deferred_list after removing the
> - * page from the split_queue, otherwise a subsequent
> - * split will see list corruption when checking the
> - * page_deferred_list.
> - */
> - list_del_init(&folio->_deferred_list);
> }
> split_queue_unlock(ds_queue);
> if (mapping) {
> @@ -3941,8 +3949,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
> shmem_uncharge(mapping->host, nr_shmem_dropped);
>
> if (!ret && is_anon)
> - remap_flags = RMP_USE_SHARED_ZEROPAGE;
> - remap_page(folio, 1 << order, remap_flags);
> + ttu_flags = (enum ttu_flags)RMP_USE_SHARED_ZEROPAGE;
> + remap_page(folio, 1 << old_order, ttu_flags);
>
> /*
> * Drop the mapping while the inode is still pinned. @folio stays
> @@ -3985,9 +3993,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
> i_mmap_unlock_read(mapping);
> out:
> xas_destroy(&xas);
> - if (order == HPAGE_PMD_ORDER)
> + if (old_order == HPAGE_PMD_ORDER)
> count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
> - count_mthp_stat(order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
> + count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
> return ret;
> }
>
> @@ -4230,35 +4238,40 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
> {
> struct deferred_split *ds_queue;
> unsigned long flags;
> - LIST_HEAD(list);
> - struct folio *folio, *next, *prev = NULL;
> - int split = 0, removed = 0;
> + struct folio *folio, *next;
> + int split = 0, i;
> + struct folio_batch fbatch;
> +
> + folio_batch_init(&fbatch);
>
> +retry:
> ds_queue = split_queue_lock_irqsave(sc->nid, sc->memcg, &flags);
> /* Take pin on all head pages to avoid freeing them under us */
> list_for_each_entry_safe(folio, next, &ds_queue->split_queue,
> _deferred_list) {
> if (folio_try_get(folio)) {
> - list_move(&folio->_deferred_list, &list);
> - } else {
> + folio_batch_add(&fbatch, folio);
> + } else if (folio_test_partially_mapped(folio)) {
> /* We lost race with folio_put() */
> - if (folio_test_partially_mapped(folio)) {
> - folio_clear_partially_mapped(folio);
> - mod_mthp_stat(folio_order(folio),
> - MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
> - }
> - list_del_init(&folio->_deferred_list);
> - ds_queue->split_queue_len--;
> + folio_clear_partially_mapped(folio);
> + mod_mthp_stat(folio_order(folio),
> + MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
> }
> + list_del_init(&folio->_deferred_list);
> + ds_queue->split_queue_len--;
> if (!--sc->nr_to_scan)
> break;
> + if (!folio_batch_space(&fbatch))
> + break;
> }
> split_queue_unlock_irqrestore(ds_queue, flags);
>
> - list_for_each_entry_safe(folio, next, &list, _deferred_list) {
> + for (i = 0; i < folio_batch_count(&fbatch); i++) {
> bool did_split = false;
> bool underused = false;
> + struct deferred_split *fqueue;
>
> + folio = fbatch.folios[i];
> if (!folio_test_partially_mapped(folio)) {
> /*
> * See try_to_map_unused_to_zeropage(): we cannot
> @@ -4281,38 +4294,27 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
> }
> folio_unlock(folio);
> next:
> + if (did_split || !folio_test_partially_mapped(folio))
> + continue;
> /*
> - * split_folio() removes folio from list on success.
> * Only add back to the queue if folio is partially mapped.
> * If thp_underused returns false, or if split_folio fails
> * in the case it was underused, then consider it used and
> * don't add it back to split_queue.
> */
> - if (did_split) {
> - ; /* folio already removed from list */
> - } else if (!folio_test_partially_mapped(folio)) {
> - list_del_init(&folio->_deferred_list);
> - removed++;
> - } else {
> - /*
> - * That unlocked list_del_init() above would be unsafe,
> - * unless its folio is separated from any earlier folios
> - * left on the list (which may be concurrently unqueued)
> - * by one safe folio with refcount still raised.
> - */
> - swap(folio, prev);
> + fqueue = folio_split_queue_lock_irqsave(folio, &flags);
> + if (list_empty(&folio->_deferred_list)) {
> + list_add_tail(&folio->_deferred_list, &fqueue->split_queue);
> + fqueue->split_queue_len++;
> }
> - if (folio)
> - folio_put(folio);
> + split_queue_unlock_irqrestore(fqueue, flags);
> }
> + folios_put(&fbatch);
>
> - spin_lock_irqsave(&ds_queue->split_queue_lock, flags);
> - list_splice_tail(&list, &ds_queue->split_queue);
> - ds_queue->split_queue_len -= removed;
> - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags);
> -
> - if (prev)
> - folio_put(prev);
> + if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) {
> + cond_resched();
> + goto retry;
> + }
>
> /*
> * Stop shrinker if we didn't split any page, but the queue is empty.
> --
> 2.53.0
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 6.18.y 3/6] mm/huge_memory: change folio_split_supported() to folio_check_splittable()
2026-09-03 18:26 ` [PATCH 6.18.y 1/6] mm: thp: introduce folio_split_queue_lock and its variants Sasha Levin
2026-09-03 18:26 ` [PATCH 6.18.y 2/6] mm: thp: use folio_batch to handle THP splitting in deferred_split_scan() Sasha Levin
@ 2026-09-03 18:26 ` Sasha Levin
2026-09-03 18:26 ` [PATCH 6.18.y 4/6] mm/huge_memory: replace can_split_folio() with direct refcount calculation Sasha Levin
` (2 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: Sasha Levin @ 2026-09-03 18:26 UTC (permalink / raw)
To: stable
Cc: Zi Yan, Wei Yang, Balbir Singh, David Hildenbrand (Red Hat),
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Liam Howlett,
Lorenzo Stoakes, Miaohe Lin, Naoya Horiguchi, Nico Pache,
Ryan Roberts, Andrew Morton, Sasha Levin
From: Zi Yan <ziy@nvidia.com>
[ Upstream commit bdd0d69a32c2aa6437d23e35acc705758b835a75 ]
Patch series "Improve folio split related functions", v4.
This patchset improves several folio split related functions to avoid
future misuse. The changes are:
1. Consolidated folio splittable checks by moving truncated folio check,
huge zero folio check, and writeback folio check into
folio_split_supported(). Changed the function return type. Renamed it
to folio_check_splittable() for clarification.
2. Replaced can_split_folio() with open coded folio_expected_ref_count()
and folio_ref_count() and introduced folio_cache_ref_count().
3. Changed min_order_for_split() to always return an order.
4. Fixed folio split stats counting.
Motivation
==========
This is based on Wei's observation[1] and solves several potential
issues:
1. Dereferencing NULL folio->mapping in try_folio_split_to_order() if it
is called on truncated folios.
2. Not handling of negative return value of min_order_for_split() in
mm/memory-failure.c
There is no bug in the current code.
This patch (of 4):
folio_split_supported() used in try_folio_split_to_order() requires
folio->mapping to be non NULL, but current try_folio_split_to_order() does
not check it. There is no issue in the current code, since
try_folio_split_to_order() is only used in truncate_inode_partial_folio(),
where folio->mapping is not NULL.
To prevent future misuse, move folio->mapping NULL check (i.e., folio is
truncated) into folio_split_supported(). Since folio->mapping NULL check
returns -EBUSY and folio_split_supported() == false means -EINVAL, change
folio_split_supported() return type from bool to int and return error
numbers accordingly. Rename folio_split_supported() to
folio_check_splittable() to match the return type change.
While at it, move is_huge_zero_folio() check and folio_test_writeback()
check into folio_check_splittable() and add kernel-doc.
Remove all warnings inside folio_check_splittable() and give warnings
in __folio_split() instead, so that bool warns parameter can be removed.
Link: https://lkml.kernel.org/r/20251126210618.1971206-1-ziy@nvidia.com
Link: https://lkml.kernel.org/r/20251126210618.1971206-2-ziy@nvidia.com
Signed-off-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Acked-by: Balbir Singh <balbirs@nvidia.com>
Acked-by: David Hildenbrand (Red Hat) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
include/linux/huge_mm.h | 6 ++--
mm/huge_memory.c | 72 +++++++++++++++++++++++------------------
2 files changed, 43 insertions(+), 35 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 240cbc6764809..9bafcdb4d66ac 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -369,8 +369,8 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
unsigned int new_order);
int min_order_for_split(struct folio *folio);
int split_folio_to_list(struct folio *folio, struct list_head *list);
-bool folio_split_supported(struct folio *folio, unsigned int new_order,
- bool uniform_split, bool warns);
+int folio_check_splittable(struct folio *folio, unsigned int new_order,
+ bool uniform_split);
int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
struct list_head *list);
/*
@@ -390,7 +390,7 @@ int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
static inline int try_folio_split_to_order(struct folio *folio,
struct page *page, unsigned int new_order)
{
- if (!folio_split_supported(folio, new_order, false, /* warns= */ false))
+ if (folio_check_splittable(folio, new_order, false))
return split_huge_page_to_list_to_order(&folio->page, NULL,
new_order);
return folio_split(folio, new_order, page, NULL);
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 27d97afd22be1..dc53a92308513 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3603,15 +3603,40 @@ static int __split_unmapped_folio(struct folio *folio, int new_order,
return ret;
}
-bool folio_split_supported(struct folio *folio, unsigned int new_order,
- bool uniform_split, bool warns)
+/**
+ * folio_check_splittable() - check if a folio can be split to a given order
+ * @folio: folio to be split
+ * @new_order: the smallest order of the after split folios (since buddy
+ * allocator like split generates folios with orders from @folio's
+ * order - 1 to new_order).
+ * @uniform_split: perform uniform split or not (non-uniform split)
+ *
+ * folio_check_splittable() checks if @folio can be split to @new_order using
+ * the requested split method. The truncated folio check must come first.
+ *
+ * Context: folio must be locked.
+ *
+ * Return: 0 - @folio can be split to @new_order, otherwise an error number is
+ * returned.
+ */
+int folio_check_splittable(struct folio *folio, unsigned int new_order,
+ bool uniform_split)
{
+ VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
+ /*
+ * Folios that just got truncated cannot get split. Signal to the
+ * caller that there was a race.
+ *
+ * TODO: this will also currently refuse folios without a mapping in the
+ * swapcache (shmem or to-be-anon folios).
+ */
+ if (!folio->mapping && !folio_test_anon(folio))
+ return -EBUSY;
+
if (folio_test_anon(folio)) {
/* order-1 is not supported for anonymous THP. */
- VM_WARN_ONCE(warns && new_order == 1,
- "Cannot split to order-1 folio");
if (new_order == 1)
- return false;
+ return -EINVAL;
} else if (!uniform_split || new_order) {
if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) &&
!mapping_large_folio_support(folio->mapping)) {
@@ -3632,9 +3657,7 @@ bool folio_split_supported(struct folio *folio, unsigned int new_order,
* case, the mapping does not actually support large
* folios properly.
*/
- VM_WARN_ONCE(warns,
- "Cannot split file folio to non-0 order");
- return false;
+ return -EINVAL;
}
}
@@ -3647,12 +3670,16 @@ bool folio_split_supported(struct folio *folio, unsigned int new_order,
* here.
*/
if ((!uniform_split || new_order) && folio_test_swapcache(folio)) {
- VM_WARN_ONCE(warns,
- "Cannot split swapcache folio to non-0 order");
- return false;
+ return -EINVAL;
}
- return true;
+ if (is_huge_zero_folio(folio))
+ return -EINVAL;
+
+ if (folio_test_writeback(folio))
+ return -EBUSY;
+
+ return 0;
}
/*
@@ -3702,31 +3729,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
goto out;
}
- /*
- * Folios that just got truncated cannot get split. Signal to the
- * caller that there was a race.
- *
- * TODO: this will also currently refuse shmem folios that are in the
- * swapcache.
- */
- if (!is_anon && !folio->mapping)
- return -EBUSY;
-
if (new_order >= old_order) {
ret = -EINVAL;
goto out;
}
- if (is_huge_zero_folio(folio)) {
- pr_warn_ratelimited("Called split_huge_page for huge zero page\n");
- return -EBUSY;
- }
-
- if (folio_test_writeback(folio))
- return -EBUSY;
-
- ret = folio_split_supported(folio, new_order, uniform_split,
- /* warns = */ false) ? 0 : -EINVAL;
+ ret = folio_check_splittable(folio, new_order, uniform_split);
if (ret) {
VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
goto out;
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 6.18.y 4/6] mm/huge_memory: replace can_split_folio() with direct refcount calculation
2026-09-03 18:26 ` [PATCH 6.18.y 1/6] mm: thp: introduce folio_split_queue_lock and its variants Sasha Levin
2026-09-03 18:26 ` [PATCH 6.18.y 2/6] mm: thp: use folio_batch to handle THP splitting in deferred_split_scan() Sasha Levin
2026-09-03 18:26 ` [PATCH 6.18.y 3/6] mm/huge_memory: change folio_split_supported() to folio_check_splittable() Sasha Levin
@ 2026-09-03 18:26 ` Sasha Levin
2026-09-03 18:26 ` [PATCH 6.18.y 5/6] migrate: replace RMP_ flags with TTU_ flags Sasha Levin
2026-09-03 18:26 ` [PATCH 6.18.y 6/6] mm/huge_memory: use folio's memcg inside __folio_split() Sasha Levin
4 siblings, 0 replies; 8+ messages in thread
From: Sasha Levin @ 2026-09-03 18:26 UTC (permalink / raw)
To: stable
Cc: Zi Yan, David Hildenbrand (Red Hat), Wei Yang, Balbir Singh,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Liam Howlett,
Lorenzo Stoakes, Miaohe Lin, Naoya Horiguchi, Nico Pache,
Ryan Roberts, Andrew Morton, Sasha Levin
From: Zi Yan <ziy@nvidia.com>
[ Upstream commit 5842bcbfc316738cbfcbdb4def5a7592aa03ebf2 ]
can_split_folio() is just a refcount comparison, making sure only the
split caller holds an extra pin. Open code it with
folio_expected_ref_count() != folio_ref_count() - 1. For the extra_pins
used by folio_ref_freeze(), add folio_cache_ref_count() to calculate it.
Also replace folio_expected_ref_count() with folio_cache_ref_count() used
by folio_ref_unfreeze(), since they are returning the same values when a
folio is frozen and folio_cache_ref_count() does not have unnecessary
folio_mapcount() in its implementation.
Link: https://lkml.kernel.org/r/20251126210618.1971206-3-ziy@nvidia.com
Signed-off-by: Zi Yan <ziy@nvidia.com>
Suggested-by: David Hildenbrand (Red Hat) <david@kernel.org>
Reviewed-by: Wei Yang <richard.weiyang@gmail.com>
Acked-by: David Hildenbrand (Red Hat) <david@kernel.org>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
include/linux/huge_mm.h | 1 -
mm/huge_memory.c | 35 ++++++++++++-----------------------
mm/vmscan.c | 3 ++-
3 files changed, 14 insertions(+), 25 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 9bafcdb4d66ac..b55a4a342f578 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -364,7 +364,6 @@ unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long add
unsigned long len, unsigned long pgoff, unsigned long flags,
vm_flags_t vm_flags);
-bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins);
int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
unsigned int new_order);
int min_order_for_split(struct folio *folio);
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index dc53a92308513..7df0a1380738c 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3326,23 +3326,6 @@ static void lru_add_split_folio(struct folio *folio, struct folio *new_folio,
}
}
-/* Racy check whether the huge page can be split */
-bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins)
-{
- int extra_pins;
-
- /* Additional pins from page cache */
- if (folio_test_anon(folio))
- extra_pins = folio_test_swapcache(folio) ?
- folio_nr_pages(folio) : 0;
- else
- extra_pins = folio_nr_pages(folio);
- if (pextra_pins)
- *pextra_pins = extra_pins;
- return folio_mapcount(folio) == folio_ref_count(folio) - extra_pins -
- caller_pins;
-}
-
static bool page_range_has_hwpoisoned(struct page *page, long nr_pages)
{
for (; nr_pages; page++, nr_pages--)
@@ -3715,7 +3698,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
- int extra_pins;
int nr_shmem_dropped = 0;
enum ttu_flags ttu_flags = 0;
pgoff_t end = 0;
@@ -3803,7 +3785,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
* Racy check if we can split the page, before unmap_folio() will
* split PMDs
*/
- if (!can_split_folio(folio, 1, &extra_pins)) {
+ if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1) {
ret = -EAGAIN;
goto out_unlock;
}
@@ -3827,7 +3809,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
/* Prevent deferred_split_scan() touching ->_refcount */
ds_queue = folio_split_queue_lock(folio);
- if (folio_ref_freeze(folio, 1 + extra_pins)) {
+ if (folio_ref_freeze(folio,
+ (folio_test_anon(folio) &&
+ !folio_test_swapcache(folio) ?
+ 0 : folio_nr_pages(folio)) + 1)) {
struct swap_cluster_info *ci = NULL;
struct lruvec *lruvec;
int expected_refs;
@@ -3896,7 +3881,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
next = folio_next(new_folio);
- expected_refs = folio_expected_ref_count(new_folio) + 1;
+ expected_refs = (folio_test_anon(new_folio) &&
+ !folio_test_swapcache(new_folio) ?
+ 0 : folio_nr_pages(new_folio)) + 1;
folio_ref_unfreeze(new_folio, expected_refs);
lru_add_split_folio(folio, new_folio, lruvec, list);
@@ -3936,7 +3923,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
* Otherwise, a parallel folio_try_get() can grab @folio
* and its caller can see stale page cache entries.
*/
- expected_refs = folio_expected_ref_count(folio) + 1;
+ expected_refs = (folio_test_anon(folio) &&
+ !folio_test_swapcache(folio) ?
+ 0 : folio_nr_pages(folio)) + 1;
folio_ref_unfreeze(folio, expected_refs);
unlock_page_lruvec(lruvec);
@@ -4465,7 +4454,7 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start,
* can be split or not. So skip the check here.
*/
if (!folio_test_private(folio) &&
- !can_split_folio(folio, 0, NULL))
+ folio_expected_ref_count(folio) != folio_ref_count(folio))
goto next;
if (!folio_trylock(folio))
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 95b1179a14e78..6d563563cecfd 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1317,7 +1317,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
goto keep_locked;
if (folio_test_large(folio)) {
/* cannot split folio, skip it */
- if (!can_split_folio(folio, 1, NULL))
+ if (folio_expected_ref_count(folio) !=
+ folio_ref_count(folio) - 1)
goto activate_locked;
/*
* Split partially mapped folios right away.
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 6.18.y 5/6] migrate: replace RMP_ flags with TTU_ flags
2026-09-03 18:26 ` [PATCH 6.18.y 1/6] mm: thp: introduce folio_split_queue_lock and its variants Sasha Levin
` (2 preceding siblings ...)
2026-09-03 18:26 ` [PATCH 6.18.y 4/6] mm/huge_memory: replace can_split_folio() with direct refcount calculation Sasha Levin
@ 2026-09-03 18:26 ` Sasha Levin
2026-09-03 18:26 ` [PATCH 6.18.y 6/6] mm/huge_memory: use folio's memcg inside __folio_split() Sasha Levin
4 siblings, 0 replies; 8+ messages in thread
From: Sasha Levin @ 2026-09-03 18:26 UTC (permalink / raw)
To: stable
Cc: Matthew Wilcox (Oracle), David Hildenbrand (Red Hat),
Lorenzo Stoakes, Zi Yan, Alistair Popple, Byungchul Park,
Gregory Price, Jann Horn, Joshua Hahn, Lance Yang, Liam Howlett,
Matthew Brost, Rakie Kim, Rik van Riel, Vlastimil Babka,
Ying Huang, Andrew Morton, Sasha Levin
From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
[ Upstream commit 832d95b5314eea558cf4cc9ca40db10122ce8f63 ]
Instead of translating between RMP_ and TTU_ flags, remove the RMP_ flags
and just use the TTU_ flag space; there's plenty available.
Possibly we should rename these to RMAP_ flags, and maybe even pass them
in through rmap_walk_arg, but that can be done later.
Link: https://lkml.kernel.org/r/20260109041345.3863089-3-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Acked-by: David Hildenbrand (Red Hat) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Jann Horn <jannh@google.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
include/linux/rmap.h | 9 +++------
mm/huge_memory.c | 6 +++---
mm/migrate.c | 12 ++++++------
3 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index daa92a58585d9..7afc6abe1c235 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -92,6 +92,7 @@ struct anon_vma_chain {
};
enum ttu_flags {
+ TTU_USE_SHARED_ZEROPAGE = 0x2, /* for unused pages of large folios */
TTU_SPLIT_HUGE_PMD = 0x4, /* split huge PMD if any */
TTU_IGNORE_MLOCK = 0x8, /* ignore mlock */
TTU_SYNC = 0x10, /* avoid racy checks with PVMW_SYNC */
@@ -1000,12 +1001,8 @@ int mapping_wrprotect_range(struct address_space *mapping, pgoff_t pgoff,
int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
struct vm_area_struct *vma);
-enum rmp_flags {
- RMP_LOCKED = 1 << 0,
- RMP_USE_SHARED_ZEROPAGE = 1 << 1,
-};
-
-void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
+void remove_migration_ptes(struct folio *src, struct folio *dst,
+ enum ttu_flags flags);
/*
* rmap_walk_control: To control rmap traversing for specific needs
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7df0a1380738c..4febfb980e2b0 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3296,7 +3296,7 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags)
if (!folio_test_anon(folio))
return;
for (;;) {
- remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
+ remove_migration_ptes(folio, folio, TTU_RMAP_LOCKED | flags);
i += folio_nr_pages(folio);
if (i >= nr)
break;
@@ -3700,8 +3700,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
enum ttu_flags ttu_flags = 0;
- pgoff_t end = 0;
int ret;
+ pgoff_t end = 0;
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
@@ -3946,7 +3946,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
shmem_uncharge(mapping->host, nr_shmem_dropped);
if (!ret && is_anon)
- ttu_flags = (enum ttu_flags)RMP_USE_SHARED_ZEROPAGE;
+ ttu_flags = TTU_USE_SHARED_ZEROPAGE;
remap_page(folio, 1 << old_order, ttu_flags);
/*
diff --git a/mm/migrate.c b/mm/migrate.c
index e66d9cc67dda3..acf93087d5ab3 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -451,11 +451,12 @@ static bool remove_migration_pte(struct folio *folio,
* Get rid of all migration entries and replace them by
* references to the indicated page.
*/
-void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
+void remove_migration_ptes(struct folio *src, struct folio *dst,
+ enum ttu_flags flags)
{
struct rmap_walk_arg rmap_walk_arg = {
.folio = src,
- .map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE,
+ .map_unused_to_zeropage = flags & TTU_USE_SHARED_ZEROPAGE,
};
struct rmap_walk_control rwc = {
@@ -463,9 +464,9 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
.arg = &rmap_walk_arg,
};
- VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src);
+ VM_BUG_ON_FOLIO((flags & TTU_USE_SHARED_ZEROPAGE) && (src != dst), src);
- if (flags & RMP_LOCKED)
+ if (flags & TTU_RMAP_LOCKED)
rmap_walk_locked(dst, &rwc);
else
rmap_walk(dst, &rwc);
@@ -1530,8 +1531,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
rc = move_to_new_folio(dst, src, mode);
if (page_was_mapped)
- remove_migration_ptes(src, !rc ? dst : src,
- ttu ? RMP_LOCKED : 0);
+ remove_migration_ptes(src, !rc ? dst : src, ttu);
if (ttu & TTU_RMAP_LOCKED)
i_mmap_unlock_write(mapping);
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 6.18.y 6/6] mm/huge_memory: use folio's memcg inside __folio_split()
2026-09-03 18:26 ` [PATCH 6.18.y 1/6] mm: thp: introduce folio_split_queue_lock and its variants Sasha Levin
` (3 preceding siblings ...)
2026-09-03 18:26 ` [PATCH 6.18.y 5/6] migrate: replace RMP_ flags with TTU_ flags Sasha Levin
@ 2026-09-03 18:26 ` Sasha Levin
4 siblings, 0 replies; 8+ messages in thread
From: Sasha Levin @ 2026-09-03 18:26 UTC (permalink / raw)
To: stable
Cc: Zi Yan, Johannes Weiner, Baolin Wang, Lorenzo Stoakes (ARM),
Barry Song, David Hildenbrand, Dev Jain, Lance Yang,
Liam R. Howlett, Matthew Wilcox (Oracle), Ryan Roberts,
William Kucharski, Andrew Morton, Sasha Levin
From: Zi Yan <ziy@nvidia.com>
[ Upstream commit c299a2285d9d8bda4da024455de65e3d00de6f17 ]
Patch series "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to
folio's memcg", v3.
__GFP_ACCOUNT is needed for xarray node allocation accounting when
XA_FLAGS_ACCOUNT is set. Commit 7b785645e8f13 ("mm: fix page cache
convergence regression") fixed a workingset regression with it.
xas_split_alloc() does not have it and needs to be fixed.
In addition, based on Sashiko's review[1] and Johannes' confirmation[2], to
charge the right memcg, folio's memcg needs to be active during folio
split. Add that before adding __GFP_ACCOUNT.
There is no workingset convergence regression related to missing
__GFP_ACCOUNT in xas_split_alloc() and the impact to userspace should be
minor.
This patch (of 2):
During a pagecache folio split, an xarray node allocation can happen and
needs to charge at folio's memcg instead of folio split invoker's memcg,
because for example folio split can happen during reclaim and reclaim's
active memcg might not be folio's memcg. Switch to folio's memcg at the
beginning and switch back afterwards.
Link: https://lore.kernel.org/20260804-add-gfp_account-to-xas_split_alloc-v3-0-38cb3ff325c5@nvidia.com
Link: https://lore.kernel.org/20260804-add-gfp_account-to-xas_split_alloc-v3-1-38cb3ff325c5@nvidia.com
Link: https://sashiko.dev/#/patchset/20260727-add-gfp_account-to-xas_split_alloc-v1-1-9fae6bf64838%40nvidia.com?part=1 [1]
Link: https://lore.kernel.org/all/amtcBZ-_QVRgCd6b@cmpxchg.org/ [2]
Fixes: 6b24ca4a1a8d ("mm: Use multi-index entries in the page cache")
Signed-off-by: Zi Yan <ziy@nvidia.com>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: William Kucharski <william.kucharski@oracle.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
mm/huge_memory.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 4febfb980e2b0..1b4134c57d007 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3694,34 +3694,42 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
XA_STATE(xas, &folio->mapping->i_pages, folio->index);
struct folio *end_folio = folio_next(folio);
bool is_anon = folio_test_anon(folio);
+ struct mem_cgroup *memcg, *old_memcg;
struct address_space *mapping = NULL;
struct anon_vma *anon_vma = NULL;
int old_order = folio_order(folio);
struct folio *new_folio, *next;
int nr_shmem_dropped = 0;
enum ttu_flags ttu_flags = 0;
- int ret;
pgoff_t end = 0;
+ int ret;
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
ret = -EINVAL;
- goto out;
+ goto out_no_memcg;
}
if (new_order >= old_order) {
ret = -EINVAL;
- goto out;
+ goto out_no_memcg;
}
ret = folio_check_splittable(folio, new_order, uniform_split);
if (ret) {
VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
- goto out;
+ goto out_no_memcg;
}
+ /*
+ * switch to folio's memcg as xarray node allocation can happen and
+ * needs to charge to it.
+ */
+ memcg = get_mem_cgroup_from_folio(folio);
+ old_memcg = set_active_memcg(memcg);
+
if (is_anon) {
/*
* The caller does not necessarily hold an mmap_lock that would
@@ -3989,6 +3997,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
if (mapping)
i_mmap_unlock_read(mapping);
out:
+ /* restore to caller's old_memcg */
+ set_active_memcg(old_memcg);
+ mem_cgroup_put(memcg);
+out_no_memcg:
xas_destroy(&xas);
if (old_order == HPAGE_PMD_ORDER)
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread