[PATCH v2 0/2] mm: fix PMD level mTHP accounting bugs

[PATCH v2 0/2] mm: fix PMD level mTHP accounting bugs

[Nico Pache]

While running selftests I noticed the PMD level per-mTHP stats (nr_anon)
remained elevated after each run. After further investigation I noticed
this accounting error occurs for both the migration.private_anon_htlb_test
and the HMM tests.

In the HMM case this is due to folio_add_new_anon_rmap() incrementing the
mTHP stats, but never containing a corresponding decrement in
free_zone_device_folio(). We solve this by making sure to decrement the
counter when freeing device memory.

In the migration case, we are incrementing this counter without first
checking whether this folio is a hugetlb folio, which relies on a separate
accounting system. We solve this by adding the proper hugetlb check before
incrementing this counter.

With these changes in place, the two tests no longer cause elevated PMD
level accounting issues.

Co-developed-by: David Hildenbrand <david@kernel.org>
Signed-off-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>

V2 Changes:
 - add RB from Zi
 - Drop unnecessary check in __folio_migrate_folio(). HugeTLB has its own
   migration system in hugetlbfs_migrate_folio().

V1: https://lore.kernel.org/all/20260702172548.37075-1-npache@redhat.com/

Nico Pache (2):
  mm: decrement MTHP_STAT_NR_ANON in free_zone_device_folio()
  mm/migrate: exclude hugetlb folios from MTHP_STAT_NR_ANON accounting

 mm/memremap.c | 1 +
 mm/migrate.c  | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)


base-commit: e57d6e9e20b551e4910d7a6331a81775c3ad6693
-- 
2.55.0

[PATCH v2 1/2] mm: decrement MTHP_STAT_NR_ANON in free_zone_device_folio()

[Nico Pache]

When a zone device folio is mapped as anonymous, folio_add_new_anon_rmap()
increments MTHP_STAT_NR_ANON. The corresponding decrement lives in
__free_pages_prepare() in page_alloc.c, but zone device folios are freed
via free_zone_device_folio() which never calls __free_pages_prepare().
This causes nr_anon to remain permanently elevated after zone device
folios are freed.

Add the missing mod_mthp_stat() decrement to free_zone_device_folio()
so that the counter is properly balanced.

Fixes: 5d65c8d758f2 ("mm: count the number of anonymous THPs per size")
Reviewed-by: Zi Yan <ziy@nvidia.com>
Co-developed-by: David Hildenbrand <david@kernel.org>
Signed-off-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/memremap.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/memremap.c b/mm/memremap.c
index 81766d822400..accba23aef28 100644
--- a/mm/memremap.c
+++ b/mm/memremap.c
@@ -425,6 +425,7 @@ void free_zone_device_folio(struct folio *folio)
 	mem_cgroup_uncharge(folio);
 
 	if (folio_test_anon(folio)) {
+		mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, -1);
 		for (i = 0; i < nr; i++)
 			__ClearPageAnonExclusive(folio_page(folio, i));
 	}
-- 
2.55.0

[PATCH v2 2/2] mm/migrate: exclude hugetlb folios from MTHP_STAT_NR_ANON accounting

[Nico Pache]

__folio_migrate_mapping() increments MTHP_STAT_NR_ANON for the
destination folio when `folio_test_anon(folio) && folio_test_large(folio)`
is true. However, hugetlb folios satisfy both conditions despite having a
completely separate accounting system — they use hugetlb_add_anon_rmap()
which does not touch mTHP stats, and their free path also bypasses the
mTHP decrement in __free_pages_prepare().

This causes MTHP_STAT_NR_ANON to be incremented on each hugetlb
migration without a corresponding decrement, permanently inflating the
nr_anon counter.

Add a !folio_test_hugetlb() check to __folio_migrate_mapping() so that
only actual mTHP folios are counted.

Fixes: 5d65c8d758f2 ("mm: count the number of anonymous THPs per size")
Reviewed-by: Zi Yan <ziy@nvidia.com>
Co-developed-by: David Hildenbrand <david@kernel.org>
Signed-off-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
---
 mm/migrate.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index d9b23909d716..dd15a84b2a52 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -590,7 +590,8 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 		/* No turning back from here */
 		newfolio->index = folio->index;
 		newfolio->mapping = folio->mapping;
-		if (folio_test_anon(folio) && folio_test_large(folio))
+		if (folio_test_anon(folio) && folio_test_large(folio) &&
+		    !folio_test_hugetlb(folio))
 			mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1);
 		if (folio_test_swapbacked(folio))
 			__folio_set_swapbacked(newfolio);
-- 
2.55.0

Re: [PATCH v2 0/2] mm: fix PMD level mTHP accounting bugs

[Andrew Morton]

On Fri, 17 Jul 2026 00:44:58 -0600 Nico Pache <npache@redhat.com> wrote:

> While running selftests I noticed the PMD level per-mTHP stats (nr_anon)
> remained elevated after each run. After further investigation I noticed
> this accounting error occurs for both the migration.private_anon_htlb_test
> and the HMM tests.
> 
> In the HMM case this is due to folio_add_new_anon_rmap() incrementing the
> mTHP stats, but never containing a corresponding decrement in
> free_zone_device_folio(). We solve this by making sure to decrement the
> counter when freeing device memory.
> 
> In the migration case, we are incrementing this counter without first
> checking whether this folio is a hugetlb folio, which relies on a separate
> accounting system. We solve this by adding the proper hugetlb check before
> incrementing this counter.
> 
> With these changes in place, the two tests no longer cause elevated PMD
> level accounting issues.

Thanks, I've updated mm.git's mm-hotfixes-unstable branch to this
version.

Sashiko is worried about the existing code ("list corruption or a
use-after-free panic"):
	https://sashiko.dev/#/patchset/20260717064502.1980173-1-npache@redhat.com

> 
> V2 Changes:
>  - add RB from Zi
>  - Drop unnecessary check in __folio_migrate_folio(). HugeTLB has its own
>    migration system in hugetlbfs_migrate_folio().

Here's how v2 altered mm.git:

 mm/migrate.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

--- a/mm/migrate.c~b
+++ a/mm/migrate.c
@@ -624,8 +624,7 @@ static int __folio_migrate_mapping(struc
 	 */
 	newfolio->index = folio->index;
 	newfolio->mapping = folio->mapping;
-	if (folio_test_anon(folio) && folio_test_large(folio) &&
-	    !folio_test_hugetlb(folio))
+	if (folio_test_anon(folio) && folio_test_large(folio))
 		mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1);
 	folio_ref_add(newfolio, nr); /* add cache reference */
 	if (folio_test_swapbacked(folio))
_

Re: [PATCH v2 0/2] mm: fix PMD level mTHP accounting bugs

[Nico Pache]

On Fri, Jul 17, 2026 at 7:29 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Fri, 17 Jul 2026 00:44:58 -0600 Nico Pache <npache@redhat.com> wrote:
>
> > While running selftests I noticed the PMD level per-mTHP stats (nr_anon)
> > remained elevated after each run. After further investigation I noticed
> > this accounting error occurs for both the migration.private_anon_htlb_test
> > and the HMM tests.
> >
> > In the HMM case this is due to folio_add_new_anon_rmap() incrementing the
> > mTHP stats, but never containing a corresponding decrement in
> > free_zone_device_folio(). We solve this by making sure to decrement the
> > counter when freeing device memory.
> >
> > In the migration case, we are incrementing this counter without first
> > checking whether this folio is a hugetlb folio, which relies on a separate
> > accounting system. We solve this by adding the proper hugetlb check before
> > incrementing this counter.
> >
> > With these changes in place, the two tests no longer cause elevated PMD
> > level accounting issues.
>
> Thanks, I've updated mm.git's mm-hotfixes-unstable branch to this
> version.

Thanks!

>
> Sashiko is worried about the existing code ("list corruption or a
> use-after-free panic"):
>         https://sashiko.dev/#/patchset/20260717064502.1980173-1-npache@redhat.com

https://sashiko.dev/#/patchset/20260717064502.1980173-1-npache@redhat.com

I dug into this, although I'm not very familiar with device memory.

It seems we have two options:

a) free_zone_device_folio() unqueue — unqueue zone device folios at
free time, preventing the corruptions/use-after-free.

b) deferred_split_folio() zone-device guard — block at queue time;
covers all callers; stopping the shrinker from ever touching/splitting
a live device folio; makes (a) redundant for this bug, but it could
still be kept as defense.

I added a few people who may be able to help.

Thanks,
-- Nico





>
> >
> > V2 Changes:
> >  - add RB from Zi
> >  - Drop unnecessary check in __folio_migrate_folio(). HugeTLB has its own
> >    migration system in hugetlbfs_migrate_folio().
>
> Here's how v2 altered mm.git:
>
>  mm/migrate.c |    3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> --- a/mm/migrate.c~b
> +++ a/mm/migrate.c
> @@ -624,8 +624,7 @@ static int __folio_migrate_mapping(struc
>          */
>         newfolio->index = folio->index;
>         newfolio->mapping = folio->mapping;
> -       if (folio_test_anon(folio) && folio_test_large(folio) &&
> -           !folio_test_hugetlb(folio))
> +       if (folio_test_anon(folio) && folio_test_large(folio))
>                 mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1);
>         folio_ref_add(newfolio, nr); /* add cache reference */
>         if (folio_test_swapbacked(folio))
> _
>

Re: [PATCH v2 0/2] mm: fix PMD level mTHP accounting bugs

[David Hildenbrand (Arm)]

On 7/21/26 15:57, Nico Pache wrote:
> On Fri, Jul 17, 2026 at 7:29 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>>
>> On Fri, 17 Jul 2026 00:44:58 -0600 Nico Pache <npache@redhat.com> wrote:
>>
>>> While running selftests I noticed the PMD level per-mTHP stats (nr_anon)
>>> remained elevated after each run. After further investigation I noticed
>>> this accounting error occurs for both the migration.private_anon_htlb_test
>>> and the HMM tests.
>>>
>>> In the HMM case this is due to folio_add_new_anon_rmap() incrementing the
>>> mTHP stats, but never containing a corresponding decrement in
>>> free_zone_device_folio(). We solve this by making sure to decrement the
>>> counter when freeing device memory.
>>>
>>> In the migration case, we are incrementing this counter without first
>>> checking whether this folio is a hugetlb folio, which relies on a separate
>>> accounting system. We solve this by adding the proper hugetlb check before
>>> incrementing this counter.
>>>
>>> With these changes in place, the two tests no longer cause elevated PMD
>>> level accounting issues.
>>
>> Thanks, I've updated mm.git's mm-hotfixes-unstable branch to this
>> version.
> 
> Thanks!
> 
>>
>> Sashiko is worried about the existing code ("list corruption or a
>> use-after-free panic"):
>>         https://sashiko.dev/#/patchset/20260717064502.1980173-1-npache@redhat.com
> 
> https://sashiko.dev/#/patchset/20260717064502.1980173-1-npache@redhat.com
> 
> I dug into this, although I'm not very familiar with device memory.

In general: Don't let a tool waste your time on unrelated things if it wasn't
you that asked the tool for it. Someone else asked the tool to find unrelated
things.

I thought we only support anonymous THP for device-private, not device-coherent.

Maybe lib/test_hmm.c could somehow trigger it by selecting
MIGRATE_VMA_SELECT_COMPOUND?

But in migrate_device_coherent_folio() we have

	WARN_ON_ONCE(folio_test_large(folio));

So this would already be pretty broken and I wouldn't spend any more time on it.

-- 
Cheers,

David